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Abstract 

There  is  concern  within  U.S  space  and  intelligence  organizations  that  the  cur¬ 
rent  Tasking,  Processing,  Exploitation,  and  Dissemination  processes  may  be  insuf- 
hcient  to  support  current  and  future  Intelligence,  Surveillance,  and  Reconnaissance 
systems.  As  part  of  a  larger  intelligence  process,  more  detailed  analysis  becomes 
critical  to  determine  what  portions  need  to  be  improved.  This  analysis  can  be  ac¬ 
complished  by  simulation,  which  is  appropriate  due  to  the  complexity  of  the  process 
and  the  ability  to  compare  variations  in  the  process.  We  construct  a  high  level  model 
of  a  generalized  military  intelligence  process  based  in  part  on  the  Intelligence  Cy¬ 
cle  outlined  in  Joint  Publications.  Using  the  Arena®  process  oriented  simulation 
software,  our  modular  simulation  can  be  used  for  quick  turn  studies  on  changes  to 
the  process,  specihcally  with  respect  to  classical  measures  such  as  quality,  quantity, 
and  timeliness.  A  sample  study  using  the  basic  framework  of  the  intelligence  process 
with  statistical  analysis  is  also  conducted. 
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MODELING  AND  SIMULATION 
OF  THE  MILITARY  INTELLIGENCE  PROCESS 


1.  Introduction 

1.1  Background 

The  importance  of  assessing  the  information  flow  of  the  military  intelligence 
process  has  been  brought  to  light  in  recent  years.  Although,  the  need  for  this  as¬ 
sessment  is  not  new,  it  becomes  more  critical  as  both  capability  and  demand  in¬ 
creases.  According  to  the  National  Security  Strategy,  “We  must  transform  our  intel¬ 
ligence  capabilities  and  build  new  ones  to  keep  pace  with”  terrorist  and  other  threats 
[White  House,  2002:30].  Military  use  of  the  intelligence  process  is  vital  both  in  and 
of  itself  and  as  a  part  of  information  superiority  which  is  an  enabler  of  military  power 
[JP  1,  2000:IV-8,9,10].  The  intelligence  process  begins  when  a  need  for  information 
or  intelligence  is  identihed  and  encompasses  how  these  information  needs  are  met. 
As  such,  it  includes  all  of  the  satellites,  aircraft,  communications,  and  other  systems 
used  to  gather  and  transmit  data  as  well  as  the  people,  organizations,  and  resources 
involved  in  turning  raw  data  into  useful  information.  The  intelligence  process  can 
take  on  many  forms,  two  of  which  are  Task,  Process,  Exploit,  Disseminate  (TPED) 
and  Task,  Process,  Post,  Use  (TPPU). 

To  illustrate  the  TPED  and  TPPU  processes,  take  for  example  a  person  with 
a  standard  35mm  camera,  lets  call  him  Bob.  The  need  for  information  about  a 
particular  place  has  arisen.  Once  this  need  is  realized  and  it  is  determined  that 
a  picture  can  satisfy  the  need.  Bob  is  directed  or  tasked  to  go  take  a  picture  of 
this  place.  Once  Bob  has  taken  the  picture  and  returned,  the  picture  is  still  of  no 
use.  A  series  of  steps  must  be  take  to  put  the  picture  into  a  usable  form,  i.e,  the 
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picture  must  be  developed  or  processed.  Now  that  the  image  is  in  a  usable  form, 
the  remainder  of  the  intelligence  process  can  be  carried  out  in  several  ways.  For  the 
TPED  approach,  an  analyst  would  take  additional  steps  to  exploit  the  picture,  e.g., 
mark  on  it  to  indicate  important  aspects  and  add  notes  to  describe  what  those  aspects 
are.  Once  this  is  complete,  the  picture  with  the  markings  and  additional  information 
would  be  sent  or  disseminated  to  the  person  or  organization  that  needed  the  picture. 
Alternatively,  the  TPPU  approach,  would  bypass  the  exploitation  up  front  and  send 
or  post  the  picture  to  a  web  page.  Then  the  person  or  organization  that  needed  the 
picture  could  retrieve  it  and  use  it  without  the  overhead  of  exploitation.  Although 
this  is  a  simplihed  example,  it  illustrates  how  TPED  and  TPPU  can  differ.  In 
reality,  the  intelligence  process  is  more  complex  and  is  dependent  on  several  systems, 
organizations,  and  user  requirements. 

1.2  Problem  Statement 

The  National  Security  Space  Architect  (NSSA)  has  indicated  that  the  TPED 
and/or  TPPU  processes  may  be  insufficient  to  support  current  and  future  ISR  sys¬ 
tems.  If  this  is  the  case,  more  detailed  analysis  of  the  intelligence  process  becomes 
critical  to  determine  what  portions  need  to  be  improved.  This  analysis  could  be  ac¬ 
complished  several  ways  such  as  detailed  statistical  analysis,  an  analytical  queueing 
network,  or  simulation.  Simulation  is  an  appropriate  tool  due  to  the  complexity  of 
the  process  and  the  ability  to  compare  multiple  ways  of  implementing  the  process. 
In  order  to  use  simulation,  an  appropriate  end-to-end  model  of  the  process  is  re¬ 
quired,  and  NSSA  has  indicated  that  such  a  model  for  comparing  various  forms  of 
the  intelligence  process  does  not  exist.  NSSA  has  identihed  the  need  for  an  appro¬ 
priate  end-to-end  model  of  the  intelligence  process  to  provide  quick  turn  analysis  of 
the  impacts  of  changes  to  the  systems  on  various  mission  areas  such  as  Intelligence, 
Surveillance,  and  Reconnaissance  (ISR) [NSSA,  2003]. 
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1.3  Research  Objective 


The  objective  is  to  develop  an  Arena®  model  of  a  generalized  national  or  mil¬ 
itary  intelligence  process.  The  model  needs  to  be  detailed  enough  to  examine  the 
various  interactions  and  resources  and  their  impact  on  fulhlling  various  missions,  yet 
flexible  enough  to  allow  simulation  of  TPED,  TPPU,  or  other  hybrid  implementa¬ 
tions  of  the  intelligence  process.  Moreover,  the  implementation  of  the  model  in  Arena 
should  allow  quick  turn  analysis.  Part  of  this  is  the  development  of  appropriate  mea¬ 
sures  of  effectiveness  and  performance.  One  important  measure  is  information  needs 
satisfaction.  Information  needs  satisfaction  encompasses  other  classical  intelligence 
measures  such  as  quality,  quantity,  and  timeliness.  It  should  also  encompass  the  cor¬ 
rect  generation  of  intelligence  requirements  that  are  passed  through  the  remainder 
of  the  intelligence  process  which  is  beyond  the  scope  of  this  model. 

1.4  Research  Focus 

This  study  will  focus  on  providing  a  basic  framework  for  national  or  military 
intelligence  process  analysis.  Determining  the  appropriate  level  of  detail  for  modeling 
various  portions  of  the  intelligence  process  is  critical.  Rather  than  focus  on  minute 
details  in  all  areas,  the  focus  will  be  on  a  top  level  model  with  additional  detail  where 
needed.  Understanding  previous  models  and  simulations  of  the  intelligence  process 
will  provide  a  background  for  model  development  and  abstraction  of  the  real  world 
process.  Comparing  prior  work  with  the  intended  application  of  the  framework 
will  reveal  areas  where  improvement  and  new  development  is  needed.  This  focus 
should  enable  development  of  the  flexibility  to  model  multiple  implementations  of 
the  intelligence  process  and  allow  quick  turn  analysis  to  be  completed. 
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1 . 5  Methodology 


In  developing  a  model,  it  is  necessary  to  ensure  that  the  model  is  structured 
for  specihc  measures  of  interest.  Those  measures  should  provide  the  required  infor¬ 
mation  for  the  intended  uses  of  the  model.  Four  measures  of  interest  for  analyzing 
the  intelligence  process  are  quality,  quantity,  timeliness,  and  information  needs  sat¬ 
isfaction  (QQTI).  The  first  three  measures,  quality,  quantity,  and  timeliness  (QQT), 
have  historically  been  used  when  assessing  various  aspects  of  the  intelligence  process. 
As  such  previous  models  may  be  designed  to  provide  those  quantitative  measures 
for  a  specific  implementation  of  the  intelligence  process.  Where  applicable  some  of 
the  concepts  in  these  models  may  be  used  in  model  development.  More  importantly, 
examination  of  open  source  and  unclassihed  descriptions  of  the  intelligence  process 
will  be  the  primary  source  for  model  development.  In  addition  to  basic  descriptions, 
factors  that  influence  the  process  and  consequently  the  QQTI  measures  will  be  taken 
into  account  based  on  subject  matter  expert  (SME)  discussions.  The  use  of  Arena  to 
model  the  military  intelligence  process  in  a  modularized  design  should  allow  the  £- 
delity  of  individual  portions  of  the  model  to  be  easily  increased  and  allow  simulation 
studies  to  be  easily  accomplished.  Further  detail  will  be  given  in  Chapter  3. 

1.6  Assumptions  and  Limitations 

The  nature  of  developing  a  model  dictates  that  some  assumptions  be  built  in 
from  the  beginning.  One  of  those  assumptions  is  the  level  of  abstraction  from  the  real 
world.  Based  on  the  intended  uses  of  the  model,  a  moderate  level  of  detail  should 
be  sufEcient  for  this  study.  That  is,  comparing  the  impact  of  changes  to  various 
portions  of  the  intelligence  process  for  quick  look  studies  does  not  require  the  detail 
of  engineering  level  models.  Furthermore,  the  moderate  level  of  detail  extends  to  all 
portions  of  the  intelligence  process  model  including  an  embedded  communications 
model.  By  maintaining  the  focus  of  building  a  top  level  framework  for  simulation 


1-4 


studies,  only  open  source  and  unclassified  information  will  be  required  to  develop 
the  structure  of  the  model.  Additional  hypothetical  data  can  be  used  to  populate 
the  model  for  testing  the  implementation  in  Arena,  but  a  well  defined  scenario  and 
review  by  SMEs  will  be  needed  for  verification  and  validation  (V&V).  Since  the 
model  is  only  an  abstraction  of  the  real  world,  it  can  be  used  to  assess  the  impact 
of  altering  a  portion  of  the  process  and  provide  insight  into  how  that  change  might 
affect  the  process  in  real  life  without  actually  altering  the  process. 

1 . 7  Preview 

This  thesis  contains  five  chapters.  This  chapter.  Introduction,  contains  back¬ 
ground  information  and  development  of  the  research  focus.  The  second  chapter. 
Literature  Review,  discusses  the  intelligence  process  in  more  detail  and  examines 
previous  work  in  modeling  and  assessing  the  intelligence  process.  The  third  chapter. 
Methodology,  discusses  the  Arena  model  development  of  a  generalized  intelligence 
process.  The  fourth  chapter.  Results  and  Analysis,  presents  the  results  of  simulation 
runs  and  statistical  analysis  of  the  simulation  output.  The  fifth  chapter.  Conclu¬ 
sions,  presents  insights  and  conclusions  based  on  the  research  and  recommendations 
for  further  study. 
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2.  Literature  Review 


2. 1  Introduction 

In  order  to  simulate  the  military  intelligence  process,  a  clear  understanding  of 
the  process  is  needed.  Examining  published  descriptions  in  addition  to  prior  models 
and  simulations  of  the  intelligence  process  will  provide  information  needed  to  gain 
that  understanding.  This  chapter  will  examine  the  intelligence  process  as  described 
in  Joint  Pub  2-0,  Doctrine  for  Intelligence  Support  to  Joint  Operations  [JP  2-0,  2000] 
and  how  Tasking,  Processing,  Exploitation  &  Dissemination  (TPED)  and  Tasking, 
Posting,  Processing  and  Using  (TPPU)  fit  into  that  process.  The  understanding 
gained  from  this  examination  will  help  to  ensure  that  a  model  of  the  process  ac¬ 
curately  represents  the  documented  process.  Furthermore,  this  chapter  will  explore 
other  models  and  simulations  with  various  properties  and  levels  of  abstraction  that 
have  been  used  to  assess  portions  of  the  intelligence  process.  Examination  of  such 
prior  work  allows  greater  understanding  of  how  others  have  abstracted  the  real  pro¬ 
cess  for  their  purposes.  However,  documentation  of  the  process  and  prior  work  will 
not  be  sufficient  in  themselves  and  discussions  with  Subject  Matter  Experts  (SME) 
will  also  be  required  for  model  development.  The  measures  of  interest  for  assessing 
the  process  are  also  key  for  developing  a  model  and  as  such  will  be  described.  Finally, 
this  chapter  will  contain  a  detailed  summary  of  how  each  of  the  items  tie  together 
for  the  development  of  a  new  simulation  model. 

2.2  Basic  Process  Description 

2.2.1  The  Intelligence  Cycle 

The  intelligence  process  is  described  in  Joint  Publication  2-0,  Doctrine  for  In¬ 
telligence  Support  to  Joint  Operations,  as  the  intelligence  cycle.  The  intelligence  cy¬ 
cle  has  six  phases.  Planning  and  Direction,  Collection,  Processing  and  Exploitation, 
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Figure  2.1  The  Intelligence  Cycle[JP  2-0,  2000:11-1] 


Analysis  and  Production,  Dissemination  and  Integration,  and  Mission  Evaluation 
and  Feedback  (see  Figure  2.1)  [JP  2-0,  2000:11-1]. 

The  first  phase.  Planning  and  Direction,  involves  planning  for  future  contin¬ 
gencies  in  theaters  and  determining  what  resources  might  be  required  for  those  con¬ 
tingencies.  A  small  number  of  Priority  Information  Requests  (PIRs)  are  created 
for  critical  information  needs  and  are  ranked  in  order  of  importance  to  the  com¬ 
mander  [JP  2-0,  2000:11-3].  Intelligence  staff  examine  the  PIRs  of  the  commander 
and  other  units  to  determine  specific  information  that  can  be  used  to  answer  the 
PIRs,  resolve  resource  conflicts,  and  remove  redundant  requests  [JP  2-0,  2000:11-4]. 
Existing  information  and  previously  scheduled  information  gathering  is  used  when 
possible,  but  the  remaining  information  requirements  are  turned  into  requests  for 
information  (RFIs)  which  can  lead  to  either  production  or  collection  requirements 
[JP  2-0,  2000:11-4].  From  those  collection  requirements,  a  collection  plan  is  devel¬ 
oped  to  prioritize  the  requirements  and  direct  them  to  to  the  organizations  best 
suited  to  meet  the  requirements  [JP  2-0,  2000:11-5]. 
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The  second  phase,  Collection,  carries  out  the  collection  plan.  Simply  put,  col¬ 
lection  is  the  gathering  of  information.  In  this  phase  organizations  or  agencies  which 
operate  collection  assets  such  as  satellites  or  surveillance  equipment  would  task  those 
assets  to  gather  information  at  specihed  times  and  places.  The  means  and  meth¬ 
ods  of  collection  are  highly  dependant  on  the  source  of  the  information  which  are 
generally  categorized  into  various  intelligence  disciplines.  Six  overarching  disciplines 
are  Imagery  Intelligence  (IMINT),  Human  Intelligence  (HUMINT),  Signals  Intelli¬ 
gence  (SIGINT),  Measurement  and  Signature  Intelligence  (MASINT),  Open-Source 
Intelligence  (OSINT),  Technical  Intelligence  (TECHINT),  and  Counterintelligence 
(Cl)  [JP  2-0,  2000:11-3].  These  disciplines  can  be  broken  down  further  based  on  the 
methods  and  technologies  used  for  collection.  Each  source  of  information  generates 
different  kinds  of  data  which,  when  collected,  often  cannot  be  immediately  used  and 
must  therefore  go  through  some  processing. 

The  third  phase.  Processing  and  Exploitation,  takes  the  raw  data  gathered 
during  the  collection  phase  and  transforms  it  so  that  it  can  be  used  for  analysis 
and  production  [JP  2-0,  2000:11-7].  The  amount  and  type  of  work  involved  depends 
on  the  type  of  intelligence  that  has  been  gathered.  It  could  involve  image  stor¬ 
age  and  conversion,  him  development,  document  translation,  or  report  generation 
[JP  2-01,  1996:111-26-29].  In  some  cases,  the  differences  between  processing  and  ex¬ 
ploitation  are  clear.  For  example  imagery  processing  might  be  conversion  of  data 
from  satellites  into  images  usable  by  a  person  while  imagery  exploitation  would  in¬ 
volve  marking  up  the  image  to  indicate  important  features  or  related  information. 

The  fourth  phase.  Analysis  and  Production,  uses  processed  and/or  exploited  in¬ 
formation  to  generate  intelligence  products  to  meet  PIRs  and  RFIs  [JP  2-0,  2000:11- 
8].  The  intelligence  products  are  usually  categorized  by  their  primary  use:  indica¬ 
tions  and  warning,  current  intelligence,  general  military  intelligence,  target  intelli¬ 
gence,  scientihc  and  technical  intelligence,  and  counterintelligence  [JP  2-0,  2000:11- 
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10-12],  These  categories  may  overlap  and  are  not  confined  to  any  particular  source 
of  intelligence. 

The  fifth  phase,  Dissemination  and  Integration,  involves  sending  intelligence 
products  to  the  user  and  the  user  using  those  intelligence  products[JP  2-0,  2000:11- 
12],  This  phase  of  the  intelligence  cycle  is  heavily  dependent  on  communications 
systems  due  to  the  likelihood  of  significant  geographic  separation  between  an  in¬ 
telligence  production  center  and  the  user.  The  dissemination  of  information  can 
be  either  “pushed”  to  the  used  to  answer  a  request  or  “pulled”  by  the  user  from 
databases  and  other  centralized  sources  of  information[JP  2-01,  1996:111-41],  The 
“pull”  method  has  the  potential  to  save  time  due  to  the  involvement  of  fewer  people 
and  organizations,  but  it  is  dependent  on  the  required  information  being  available. 
For  the  information  to  be  available,  it  must  have  been  previously  collected  and  an 
appropriate  amount  of  processing  been  accomplished.  Additionally,  the  user  that  is 
pulling  the  information  must  have  access  to  the  databases  or  repositories  that  con¬ 
tain  the  information.  Once  the  user  has  the  intelligence  product,  integration  or  use 
of  the  information  may  still  require  some  intelligence  resources  if  it  is  incomplete  or 
needs  additional  explanation. 

The  last  phase.  Mission  Evaluation  and  Feedback,  is  integral  to  all  of  the  other 
phases  and  is  appropriately  placed  in  the  center  of  Figure  2.1.  It  is  not  conducted 
independently  but  must  be  accomplished  throughout  each  phase  to  ensure  that  the 
process  is  working  as  expected.  Qualitative  attributes  that  are  used  to  evaluate  the 
quality  of  intelligence  are  timeliness,  accuracy,  usability,  completeness,  relevance, 
objectiveness,  and  availability  [JP  2-0,  2000:11-14],  Of  these,  timeliness  can  be  eas¬ 
ily  quantified  by  comparing  the  time  of  delivery  with  the  time  of  need.  Timely 
intelligence  is  delivered  to  the  user  before  the  time  that  it  is  needed.  An  example 
of  calculating  timeliness  focused  on  collection  is  given  in  Figure  2.2.  The  other  at¬ 
tributes  may  be  difficult  to  quantify,  but  they  are  nonetheless  vital  when  evaluating 
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COLLECTION  TIMELINE 
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Timeliness  =  T+T1+T2+T3+T4+T5+T6+T7 

If  timeliness  is  less  than  or  equal  to  NLT  time  information  required,  then  collection  effort  is  of  value 
If  timeliness  is  greater  than  NLT  time  information  required,  then  collection  effort  is  of  NO  value 


Figure  2.2  Example  of  how  to  determine  timeliness  [JP  2-01,  1996:111-19]. 


the  overall  quality  of  intelligence.  When  any  of  these  attributes  are  lacking,  they 
indicate  problems  within  the  intelligence  cycle. 


2.2.2  Variation  of  Process  Implementation  (TPED  vs.  TPPU) 

The  intelligence  cycle  described  in  Section  2.2.1  provides  a  good  general  de¬ 
scription  of  the  intelligence  process,  but  does  not  capture  much  of  the  variation  that 
occurs  when  the  intelligence  process  is  implemented.  Two  primary  implementations 
of  the  intelligence  process  are  Task,  Process,  Exploit,  Disseminate  (TPED)  and  Task, 
Process,  Post,  Use  (TPPU).  Although  not  the  only  possibilities,  they  are  two  current 
philosophies  of  how  to  accomplish  the  intelligence  process. 

The  hrst  two  parts  of  both  TPED  and  TPPU  are  tasking  and  processing. 
This  should  be  the  case  for  any  implementation  of  the  intelligence  process.  Tasking 
of  collection  assets  is  a  necessary  component  since  information  must  be  collected 
in  order  to  use  it.  Furthermore,  processing  cannot  be  omitted  since  the  raw  data 
that  is  collected  must  be  put  into  a  usable  format.  The  remainder  of  the  process 
varies  in  how  much  additional  effort  is  put  into  adding  relevant  information  to  the 
collected  data,  when  the  data  is  sent  to  the  user,  and  how  the  user  acquires  the 
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information.  The  TPED  process  essentially  follows  the  intelligence  cycle  from  phase 
to  phase  beginning  with  the  tasking  of  collection  resources.  The  important  aspects 
of  the  TPED  process  are  that  processing  and  exploitation  are  completed  to  generate 
an  intelligence  product  that  is  then  disseminated  to  the  user.  The  TPPU  process 
differs  in  that  data  is  posted  to  centralized  libraries  or  databases  at  several  places 
throughout  the  intelligence  process.  Data  can  be  immediately  posted  after  collection, 
processing,  and/or  exploitation.  One  beneht  of  this  method  is  that  the  user  can  then 
access  the  raw  or  processed  data  much  sooner  than  waiting  for  a  hnished  intelligence 
product. 

2.3  Assessment  Measures 

One  of  the  critical  pieces  of  information  that  is  needed  when  developing  a  model 
is  the  question  to  be  answered  or  the  problem  to  be  explored.  In  order  to  answer  the 
question,  one  must  have  a  means  of  collecting  data.  This  is  where  specihc  measures 
come  into  play.  Measures  that  are  implemented  in  a  simulation  model  should  be 
quantitative  in  nature  and  provide  sufficient  information  to  answer  the  question  or 
gain  insight  into  the  problem.  When  assessing  the  intelligence  process,  three  mea¬ 
sures  are  commonly  used:  quality,  quantity,  and  timeliness  (QQT)  of  information. 
Another  measure,  information  needs  satisfaction,  is  less  common  but  as  an  aggregate 
measure  allows  more  insight  into  the  overall  process.  Each  of  these  measures  and 
some  of  their  uses  are  outlined  in  the  following  paragraphs. 

Quality  of  information  is  often  a  qualitative  measure,  but  in  some  cases  can  be 
quantitative.  A  quantitative  example  is  image  resolution.  However,  the  image  reso¬ 
lution  alone  does  not  capture  all  aspects  of  quality,  e.g.,  an  image  of  something  that 
is  obscured  by  clouds  has  low  quality  even  if  it  has  high  resolution.  Furthermore, 
other  sources  of  information  do  not  necessarily  have  such  objectively  quantihable 
attributes.  Some  of  the  attributes  of  quality  intelligence  are  timeliness,  accuracy,  us¬ 
ability,  completeness,  relevance,  objectiveness,  and  availability  [JP  2-0,  2000:11-14]. 
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However,  timeliness  is  often  separated  from  quality  as  a  separate  measure.  Specific 
types  of  information  may  also  have  other  attributes  that  contribute  to  overall  qual¬ 
ity.  In  general,  quality  of  information  is  important,  since  wrong  or  bad  information 
can  result  in  unacceptable  loss  of  life,  especially  in  times  of  conflict.  Qualitative 
attributes  can  be  put  into  categories  such  as  “excellent,”  “good,”  or  “poor”  which 
have  a  clear  order.  Accordingly,  quality  can  also  be  represented  with  ordinal  num¬ 
bers.  For  example,  one  could  rate  the  quality  of  information  on  a  scale  from  one 
to  hve,  with  one  being  the  highest  quality.  More  detailed  methods  of  rating  quality 
could  ordinally  rank  each  of  several  attributes  and  aggregate  those  ratings  into  an 
overall  quality  rating.  Another  more  detailed  approach  to  examining  quality  is  using 
a  Knowledge  Matrix. 

Quantity  of  information  is  usually  a  quantitative  measure.  Just  as  the  number 
of  products  produced  by  a  factory  is  important,  so  is  the  number  of  information 
requests  fulhlled  through  the  intelligence  process.  The  quantity  of  information  that 
a  collection  platform,  intelligence  analyst,  or  communication  system  can  handle  is 
vital  in  determining  the  number  of  information  requests  that  the  overall  system 
can  process.  The  number  of  requests  that  can  be  answered  is  as  important  as  the 
quality  of  information.  Unanswered  requests  may  leave  a  commander  and  troops  in 
a  vulnerable  situation.  In  general,  a  large  amount  of  information  that  is  poor  is  of 
little  value  and  may  actually  be  worse  than  a  small  amount  of  good  information. 

Timeliness  of  responses  to  information  requests  is  also  a  quantitative  measure. 
As  previously  discussed  in  Section  2.2.1,  timeliness  is  used  to  determine  if  requests 
are  met  on  time.  As  a  quantitative  measure,  it  can  be  tracked  as  a  total  time  from 
request  as  in  Figure  2.2  or  as  a  difference  between  the  time  required  and  the  time 
completed.  Late  responses  may  be  as  detrimental  as  unanswered  requests.  The  later 
the  response,  the  less  useful  it  is  likely  to  be. 

Although  quality,  quantity,  and  timeliness  of  information  are  important  to 
asses  the  various  aspects  of  the  intelligence  process,  a  single  aggregated  measure  can 
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sometimes  be  more  useful.  NSSA  has  indicated  that  information  needs  satisfaction 
(INS)  is  one  such  measure  [NSSA  Meeting,  2003].  Although  it  can  be  viewed  as 
the  process  of  meeting  information  needs,  as  a  measure  it  can  describe  either  the 
proportion  of  needs  that  were  met  or  the  degree  to  which  needs  were  met.  When 
specihcally  related  to  QQT,  INS  would  simply  be  the  proportion  of  intelligence 
requests  that  meet  both  of  the  quality  and  timeliness  requirements.  Alternatively, 
INS  could  be  rated  on  a  scale  with  items  that  meet  both  of  the  quality  and  timeliness 
requirements  at  the  top  and  decreasing  scores  based  on  how  poor  or  how  late  the 
response  to  a  request  was  received.  This  would  allow  percentages  based  on  various 
levels  of  satisfaction. 

2.4  Previous  Work 

Many  models  and  simulations  have  been  used  to  assess  various  parts  of  the 
intelligence  process  in  the  past.  In  each  case,  the  designers  of  the  model  selected 
a  level  of  detail  as  well  as  one  or  more  implementations  or  architectures  of  the  in¬ 
telligence  process  such  as  TPED  or  TPPU.  Models  have  ranged  from  low  detail 
and  one  architecture  to  high  detail  and  many  architectures.  Each  position  in  the 
level  of  detail  versus  number  of  architectures  space  provides  important  insight  into 
different  problems.  Examining  models  from  each  of  the  four  corners  of  this  space 
will  provide  vital  insight  into  how  a  model  should  be  developed  for  the  problem  at 
hand.  The  hrst  corner  explored  is  that  containing  models  that  are  highly  detailed 
and  model  only  a  single  architecture.  The  second  corner  explored  is  that  contain¬ 
ing  models  that  have  a  low  amount  of  detail  and  model  only  a  single  architecture. 
The  Intelligence,  Surveillance,  Reconnaissance  -  Tasking,  Processing,  Exploitation, 
Dissemination  (ISR-TPED)  model  is  an  example  of  this  and  is  described  in  Section 
2.4.2.  The  third  corner  explored  is  that  containing  models  that  are  highly  detailed 
and  model  multiple  architectures.  The  Command,  Control,  Communications,  Com¬ 
puters,  Intelligence,  Surveillance,  and  Reconnaissance  (C4ISR)  Space  and  Missile 
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Table  2.1  Prior  models  in  the  level  of  detail  versus  number  of  architecture  space. 

Amount  of  Detail 


4f  Arch 

Low 

High 

One 

ISR-TPED 

ISR  Platform  Models 

Several 

QUICM 

COSMOS 

Operations  Simulator  (COSMOS)  formerly  known  as  Intelligence,  Surveillance  and 
Reconnaissance  Simulator  (ISRSIM)  is  an  example  of  this  and  is  described  in  Section 
2.4.3.  The  last  corner  is  that  containing  models  that  have  a  low  amount  of  detail 
and  model  multiple  architectures.  The  Quick  ISR  Concept  of  Operations  (CONOPS) 
Modeler  (QUICM)  is  an  example  of  this  and  is  described  in  Section  2.4.4.  In  Table 
2.1  some  of  these  prior  simulations  are  given  along  with  where  they  are  in  the  detail 
vs  number  of  architecture  space.  The  advantages  and  disadvantages  of  each  corner 
of  the  space  are  discussed  with  each  of  the  models. 

2.4-1  ISR  Platform  Models 

Many  low  level  (i.e,  highly  detailed)  models  exist  for  many  specihc  ISR  plat¬ 
forms.  These  allow  detailed  analysis  of  those  specihc  systems  and  their  individual 
performance  but  provide  little  to  no  detail  on  overall  impact  to  the  Intelligence 
process. 

2.4.2  ISR-TPED 

Some  higher  level  aggregate  models  exist,  but  are  usually  designed  to  examine  a 
specihc  architecture  of  the  intelligence  process.  An  example  of  this  sort  of  simulation 
is  ISR-TPED.  It  provides  a  way  to  quickly  analyze  the  TPED  architecture,  but  lacks 
in  its  ability  to  explore  other  architectures  such  as  TPPU. 

ISR-TPED  is  an  analytical  simulation  which  has  advantages  over  a  discrete 
event  simulation  (DES).  The  primary  advantage  is  runtime.  An  analytic  simulation 
signihcantly  reduces  the  time  to  examine  the  system  since  it  is  evaluated  analytically 
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and  can  essentially  be  done  in  one  pass.  However,  this  advantage  carries  a  signif¬ 
icant  disadvantage  with  it,  primarily  that  the  type  of  probability  distributions  are 
extremely  limited  due  to  the  complexity  of  computation. 

ISR-TPED  has  another  signihcant  disadvantage,  its  rigid  structure.  The  struc¬ 
ture  strictly  follows  the  Tasking,  Processing,  Exploitation,  Dissemination  path.  That 
rigid  structure  reduces  the  complexity  and  allows  the  analytic  solution  to  be  found 
easier.  However,  the  rigidity  brings  into  question  its  ability  to  model  the  real  world. 

Another  detractor  for  the  current  study  of  ISR-TPED  is  that  it  relies  on  a 
moderately  detailed  ground  picture  for  the  scenario.  The  ground  picture  allows 
one  target  per  grid-space  with  associated  terrain/clutter  type.  Sensor  performance 
must  be  calculated  by  another  simulation  called  ISR  Performance  Evaluation  Tool 
(ISR-PET),  which  is  automatically  executed  if  the  selected  combination  is  not  found 
in  a  pre-built  database.  An  additional  problem  with  ISR-PET  is  that  it  currently 
only  includes  synthetic  aperture  radar  (SAR)  performance  models.  Although  it  may 
be  possible  to  approximate  other  sensors  or  collection  assets  with  the  SAR  models, 
the  beneht  of  the  detailed  model  is  lost.  Additionally,  evaluating  the  comparative 
performance  of  various  architectures  does  not  necessarily  require  detailed  sensor 
models,  as  long  as  they  can  be  appropriately  approximated. 

The  benehts  of  a  tool  in  the  same  category  as  ISR-TPED  is  that  one  can 
assess  the  specihed  architecture  and  hnd  areas  for  more  detailed  study  or  process 
improvement. 

2.4.3  COSMOS 

At  another  extreme  are  models  that  are  very  detailed  and  allow  comparisons  of 
multiple  architectures.  An  example  of  this  is  the  COSMOS  simulation  environment. 
COSMOS  provides  engineering  level  models  for  sensors  and  other  elements.  This 
allows  for  accurate  representation  of  current  systems.  However,  it  also  makes  it  dif- 
hcult  to  evaluate  suggested  systems  when  that  level  of  detail  is  unknown.  COSMOS 
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allows  the  intelligence  architecture  to  be  modihed  so  that  comparative  analysis  can 
be  done.  This  sort  of  detailed  analysis  can  be  useful  if  a  specihc  architecture  is  known 
or  “highly  suspected”  to  provide  an  improvement  over  current  operations.  However, 
to  assess  an  architecture  that  has  unknown  impact,  this  sort  of  detailed  model  car¬ 
ries  with  it  a  large  amount  of  overhead  that  can  unduly  increase  the  cost  of  model 
development,  significantly  increase  the  required  run-time,  and  produce  an  inordinate 
amount  of  unneeded  information.  These  together  make  COSMOS  an  inappropriate 
tool  for  analysis  in  quick  turn  studies  where  such  detail  is  unnecessary. 

244  QUICM 

The  hnal  category  of  models  or  simulations  are  the  most  appropriate  for  the 
desired  purpose  of  comparative  analysis  of  various  intelligence  process  architectures. 
QUICM  provides  much  of  the  required  functionality.  First  of  all,  it  does  not  include 
detailed  sensor  models,  but  does  include  a  variety  of  information  sources.  This 
is  useful  since  it  allows  for  the  interaction  of  several  types  of  information  sources 
throughout  the  intelligence  process.  It  also  avoids  the  unnecessary  (for  our  purposes) 
overhead  of  the  detailed  scenarios  of  target  locations  and  terrain  models. 

An  additional  benefit  of  QUICM  is  that  it  allows  for  the  comparison  of  TPED 
with  TPXX  type  (i.e.,  TPPU)  architectures.  Unfortunately,  it  does  not  currently 
allow  for  hybrid  architectures,  i.e.,  all  information  follows  the  TPED  path  or  it 
follows  the  TPXX  path. 

2. 5  Summary 

The  six  phases  of  the  intelligence  cycle  (see  Figure  2.1)  provide  a  high  level 
model  of  how  intelligence  operations  are  conducted.  From  the  generation  of  informa¬ 
tion  requirements  through  dissemination  to  the  user.  More  importantly,  it  allows  us 
to  compare  TPED  and  TPPU.  TPED  closely  follows  the  intelligence  cycle  whereas 
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TPPU  allows  the  user  to  retrieve  the  raw  data  or  processed  information  from  a  cen¬ 
tralized  source  prior  to  the  generation  of  an  intelligence  product.  Once  the  process 
has  been  clearly  dehned,  measuring  its  performance  becomes  possible. 

Intelligence  operations  have  traditionally  used  quality,  quantity,  and  timeliness 
to  measure  the  performance  of  the  intelligence  process.  Each  of  these  elements 
are  vital,  but  a  high  level  aggregate  measure  can  be  more  useful  when  comparing 
overall  performance.  One  such  measure  is  information  needs  satisfaction  which  can 
be  simply  represented  as  a  proportion  of  needs  that  have  been  completely  met,  e.g., 
both  on  time  and  of  sufficient  quality.  Use  of  these  measures  allows  for  assessment 
of  the  actual  process  and  of  a  model  of  the  process. 

Modeling  and  simulation  is  commonly  used  to  assess  the  performance  of  por¬ 
tions  of  the  intelligence  process.  In  order  to  model  the  intelligence  process,  some  level 
of  abstraction  has  to  be  selected  along  with  the  particular  implementation  of  the  pro¬ 
cess.  The  models  described  in  Section  2.4  range  from  high  fidelity  to  low  hdelity  and 
range  from  a  single  element  of  the  architecture  to  multiple  implementations  of  the 
intelligence  process.  For  quick  look  studies  a  low  hdelity  or  high  level  model  that 
describes  multiple  implementations  is  ideal.  The  QUICM  model  comes  closest  to  the 
desired  product,  but  is  still  lacking  in  that  it  only  models  two  implementations  of 
the  intelligence  process.  The  ability  to  model  hybrid  implementations  allows  for  a 
better  representation  of  reality  and  analysis  of  transitional  stages  between  complete 
TPED  and  TPPU  implementations.  For  example,  such  a  transitional  stage  might  be 
altering  the  process  for  one  type  of  information  or  for  a  specihc  user  and  examining 
the  impact  of  the  overall  system.  Rather  than  modify  one  of  the  existing  models,  a 
new  model  tailored  to  the  intelligence  cycle  would  be  preferred. 
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3.  Methodology 

3. 1  Introduction 

The  primary  purpose  of  this  study  is  to  develop  a  new  simulation  model  of  the 
intelligence  process  for  use  in  quick  turn  studies.  The  top  level  focus  and  the  need 
for  relatively  short  runtime  precludes  the  inclusion  of  too  much  detail  in  the  model. 
As  such,  a  high  level  description  of  the  process  is  initially  more  benehcial  to  model 
development  than  detailed  descriptions  that  require  additional  work  to  aggregate. 
The  process  description  in  Section  2.2  provides  a  basis  for  a  conceptual  model.  An 
appropriate  simulation  environment  for  model  development  must  also  be  selected 
with  the  intended  audience  in  mind.  This  chapter  outlines  the  selection  of  the  Arena 
simulation  environment  for  model  implementation.  A  detailed  description  of  the 
model  as  implemented  is  given  in  Section  3.3.  Since  the  measures  used  to  assess 
the  process  performance  are  as  important  as  the  model  itself,  the  implementation  of 
these  measures  is  given  in  Section  3.4.  Some  of  the  steps  taken  to  validate  and  verify 
the  model  are  discussed  in  Section  3.5.  Furthermore,  some  of  the  data  that  may  be 
required  for  many  potential  simulation  studies  is  discussed  in  Section  3.6  followed 
by  a  brief  description  of  a  sample  study  setup  in  Section  3.7. 

3.2  Selection  of  Simulation  Environment 

When  developing  a  new  simulation  model,  many  options  are  available  for  a 
development  environment.  At  a  minimum,  a  simulation  programming  language  is 
desirable.  However,  the  use  of  a  visual  simulation  environment  designed  for  process 
modeling  aids  in  the  implementation  of  a  conceptual  process  model.  A  visual  envi¬ 
ronment  also  aids  in  teaching  others  how  a  model  works,  which  is  beneficial  when 
the  developer  of  the  model  may  not  be  the  only  user  of  it.  In  many  cases,  a  visual 
development  environment  may  not  be  feasible  due  to  the  scale  or  detail  required  for 
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Intelligence  Process  Model 

Based  on  die  intelligence  Cycle  from  JP  2-0  (cnapter  li) 


I  ^  Communications 

-  Contains  core  logic 
of  the  model 

-  Routes  entities  where 
they  need  to  go  next 
based  on  attributes 


^  Processing  and  Exploitation 


-  Perform  processing 

and/or  exploitati  on  if  pr - : - ; - 

required  by  user  \  Analysis  and  Production 


-  Perform  additional 

analysis  and  production  i  - : — - — - : — 

if  required  by  users  ^  Dissemination  and  Integration 


-Disseminate  to  users 

-  User  Integration  into  pr - ; - 

processes  ^Evaluation 


Evaluate/Record 
timeliness,  percent 
of  RFIs  met,  and 
other  statistics 


Figure  3.1  Top  level  of  the  Intelligence  Process  Model 


the  model.  However,  this  study  intends  to  develop  only  a  top  level  model  which 
makes  a  visual  development  environment  ideal.  Although  several  process  modeling 
environments  exist  and  would  be  adequate  for  the  purposes  of  this  study,  Arena  was 
selected  because  it  was  availaible,  appropriate,  credible,  and  widely  used. 


3.3  Intelligence  Process  Model  (IPM)  Description 


The  conceptual  model  derived  from  the  process  description  in  Section  2.2  was 
implemented  in  Arena  from  a  top-down  perspective.  The  top  level  of  the  model 
shown  in  Figure  3.1  is  composed  of  seven  submodels,  six  of  which  are  taken  directly 
from  the  Intelligence  Cycle  given  in  Figure  2.1.  The  last  submodel  is  a  Commu¬ 
nications  submodel  which  ties  all  of  the  other  submodels  together.  This  modular 
approach  allows  any  submodel  to  be  easily  modihed  or  replaced  and  keeps  the  var¬ 
ious  portions  of  the  process  distinct.  Each  of  the  submodels  is  described  in  the 
following  sections. 


1^  Planning  and  Direction 

-  Generate  InfoReq/RFI  here 
for  users 

-  Set  user  requirements 
for  priority,  timeliness, 
quality,  amount  of 
processing  required 
(e,g.,TPEDvsTPPU) 

-  Determine  overall  priority 
or  order  of  collection 
based  on  users  and  their 
requirements 

-  Perform  Libraiy  Search 


Collection  | 

-Task resources  here 
to  collect  if  required 
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3.3.1  Planning  and  Direction  Submodel 

The  Planning  and  Direction  submodel  is  modeled  as  the  beginning  of  the  intel¬ 
ligence  process  and  corresponds  directly  with  the  process  of  the  same  name  from  the 
Intelligence  Cycle  (see  Figure  2.1).  The  purpose  of  this  submodel  is  to  generate  user 
information  requirements  and  prioritize  them  globally.  As  an  additional  part  of  the 
planning  process,  users  perform  a  library  search  to  determine  if  existing  information 
may  meet  their  needs.  The  details  of  the  three  main  portions  of  this  submodel  are 
given  below. 

Prior  to  examining  the  various  portions  of  this  submodel,  it  is  important  to 
note  that  a  single  entity  type,  RFI,  is  used  throughout  the  simulation  model.  These 
entities  are  related  to  real  world  requests  for  information,  but  are  not  exactly  the 
same.  One  difference  is  that  RFIs  in  the  model  are  only  gathered  from  a  single 
information  source,  whereas  in  the  real  world  information  from  multiple  sources  may 
be  required.  The  primary  reason  for  this  difference  is  that  allowing  a  single  RFI  to 
have  multiple  sources  of  information  is  beyond  the  scope  of  this  study.  If  multiple 
source  capability  is  needed,  it  could  be  added  during  future  study.  Additionally,  the 
issue  of  how  various  high  level  questions  are  decomposed  into  smaller  questions  that 
can  be  addressed  from  a  single  source  must  be  addressed.  Another  difference  is  that 
an  RFI  in  the  model  more  closely  resembles  a  tracking  sheet  that  would  theoretically 
follow  a  real  information  request  through  the  process.  The  rationale  for  choosing 
this  abstraction  is  that  assessment  of  the  intelligence  process  at  a  top  level  does  not 
require  actual  information,  only  the  status  of  requests.  More  specifically,  only  the 
knowledge  of  required  information  quality  and  the  knowledge  of  actual  information 
quality  received  is  needed  for  assessment  of  meeting  quality  requirements.  As  such, 
each  RFI  is  assigned  various  attributes  that  hold  information  about  the  request 
requirements  and  actual  information  quality  that  are  determined  elsewhere  in  the 
model.  This  method  also  aids  in  the  generation  of  statistics  to  evaluate  the  measures 
of  performance. 
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Requirements  Planning  User  1 


Figure  3.2  User  Requirements  Planning  submodel  of  the  Planning  and  Direction 
submodel 

3. 3. 1.1  User  Requirements  Planning 

The  Planning  and  Direction  submodel  contains  hve  user  requirements  planning  sub¬ 
models.  One  of  these  submodels  is  given  in  Figure  3.2,  but  all  are  identical  with  the 
exception  of  numbering.  The  RFI  entities  are  generated  within  these  submodels. 
Each  user  has  two  RFI  generators,  one  for  standing  requirements  that  are  assumed 
to  require  collection  without  a  library  search  and  another  for  additional  requirements 
that  are  expected  to  undergo  a  library  search  before  a  collection  determination  is 
made. 

Once  the  entities  are  created,  the  attributes  of  the  requirements  they  repre¬ 
sent  are  assigned  to  them.  Specifically,  each  is  assigned  a  value  for  the  following 
attributes:  User,  Standard,  InfoSource,  TimeR,  QualR,  and  Priority.User.  The  first 
two  of  these  attributes  are  assigned  values  used  to  track  where  the  requirement  orig¬ 
inated,  whereas  the  last  four  are  assigned  values  used  to  describe  the  requirement 
itself.  A  description  of  the  purpose  of  each  of  these  attributes  is  given  in  Table  3.1. 

If  an  RFI  is  from  a  standing  requirement,  then  it  will  also  be  assigned  at¬ 
tributes  that  determine  which  steps  of  the  intelligence  process  must  be  undertaken 
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Table  3.1 

Description  of  RFI  attributes  that  describe  requirements. 

Attribute 

Description 

User 

Indicates  which  of  5  users  generated  the  requirement 

Standard 

Indicates  if  the  requirement  is  standing  (I)  or  additional  (0) 

Info  Source 

Indicates  which  one  of  13  sources  are  needed  to  satisfy  the  requirement 

TimeR 

Indicates  the  time  from  creation  that  a  requirement  needs  to  be  filled 

QualR 

Indicates  a  required  level  of  information  quality  from  1  to  5  (5  is  best) 

Priority -User 

Indicates  user  ranked  priority  of  a  requirement  from  I  to  5  (I  is  highest) 

Table  3.2  Description  of  RFI  attributes  that  describe  steps  needed  to  fulfill  a 
requirement. 


Attribute 

Description 

Collect 

Indicates  if  collection  is  (1)  or  is  not  (0)  needed 

Process 

Indicates  if  processing  is  (I)  or  is  not  (0)  needed 

Exploit 

Indicates  if  exploitation  is  (1)  or  is  not  (0)  needed 

Analyze 

Indicates  if  analysis  is  (1)  or  is  not  (0)  needed 

Produce 

Indicates  if  production  is  (I)  or  is  not  (0)  needed 

Disseminate 

Indicates  if  dissemination  is  (I)  or  is  not  (0)  needed 

Integrate 

Indicates  if  user  integration  is  (I)  or  is  not  (0)  needed 

to  fulfill  a  particular  requirement.  These  attributes,  Collect,  Process,  Exploit,  Ana¬ 
lyze,  Produce,  Disseminate,  and  Integrate,  correspond  directly  to  the  various  portions 
of  the  submodels  in  the  IPM  (see  Table  3.2).  Note  that  there  are  not  attributes  that 
correspond  to  the  Planning,  Communications,  or  Evaluation  portions  of  the  model 
since  all  entities  must  travel  through  these  portions  of  the  IPM  regardless  of  other 
steps  of  the  process  that  must  be  completed. 

If  an  RFI  is  from  an  additional  requirement,  then  it  will  be  sent  out  of  the  user 
planning  submodel  so  that  a  library  search  can  be  performed.  When  these  entities 
return  from  a  library  search,  the  results  of  the  search  are  evaluated  to  determine 
whether  or  not  the  requirement  can  be  met  with  the  information  found  or  if  new 
collection  must  be  accomplished.  In  order  for  a  requirement  to  be  met,  relevant 
information  must  be  found,  the  information  quality  requirement  must  be  met,  and 
the  age  of  the  information  must  not  be  too  old.  This  binary  determination  is  made 
by  multiplying  three  corresponding  expressions  together  that  evaluate  to  either  0  or 
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Exit  1)  Library 
Exit  2)  Planning 


Figure  3.3  Planning  portion  of  the  Planning  and  Direction  submodel 

1.  If  all  three  expressions  are  met  (i.e.,  equal  one),  then  the  Collect  attribute  will  be 
assigned  a  value  of  0.  Otherwise,  the  Collect  attribute  will  be  assigned  a  value  of  1. 
The  remainder  of  the  attributes  that  determine  what  steps  of  the  intelligence  process 
must  be  accomplished  (see  Table  3.2)  are  also  set  based  on  the  results  of  the  library- 
search,  but  are  left  up  to  the  user  to  determine  what  proportion  of  requirements  will 
need  each  step  accomplished. 

3. 3. 1.2  Overall  Planning 

Once  RFIs  are  generated  and  leave  the  Requirements  Planning  User  submodels,  they 
are  tagged  to  either  go  to  a  library  search  or  are  assigned  a  global  priority  based  on 
how  they  exited  the  submodels.  For  those  items  that  need  to  have  a  library  search 
performed,  an  attribute  named  NextStation  is  assigned  a  value  that  corresponds  to 
the  library.  The  role  of  the  NextStation  attribute  is  discussed  in  more  detail  in 
Section  3.3.7.  The  remainder  of  the  items  are  assigned  a  global  priority  attribute. 
Priority _Clobal,  that  is  dependent  upon  the  User  attribute.  The  method  of  assigning 
values  to  Priority .Clobalj  is  left  up  to  the  simulation  user,  but  in  many  cases  should 
involve  the  use  of  the  Priority^User  attribute.  As  with  the  Priority -User,  the  values 
Priority -Clobal  should  be  such  that  lower  values  have  a  higher  priority.  In  addition 
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Figure  3.4  Library  Search  portion  of  the  Planning  and  Direction  submodel 

to  assigning  the  overall  priority,  the  NextStation  attribute  is  given  a  value  of  0.  All 
entities  are  also  assigned  a  LastStation  attribute  that  corresponds  to  the  the  planning 
portion  of  the  Planning  and  Direction  submodel.  This  portion  of  the  submodel  also 
sends  and  receives  entities  from  the  Communications  submodel.  For  those  items  that 
arrive  from  a  library  search,  they  are  sent  back  to  the  user  of  origin.  Two  dispose 
blocks  are  included  in  this  portion  of  the  model  to  aid  in  verihcation.  No  entities 
should  ever  meet  the  conditions  to  be  disposed  here.  In  the  event  that  entities 
are  disposed  here,  it  would  indicate  that  the  User  attribute  has  been  assigned  an 
incorrect  value  or  the  LastStation  attribute  has  a  value  that  the  planning  portion  of 
the  model  is  not  constructed  to  handle. 

3.3. 1.3  Library  Search 

The  library  search  portion  of  the  Planning  and  Direction  submodel  has  the  primary 
purpose  of  determining  the  results  of  users  attempting  to  hnd  existing  information 
that  meets  the  needs  of  their  requirements.  When  RFIs  enter  the  library,  they 
are  immediately  put  in  a  low  value  hrst  (LVF)  queue  based  on  the  Priority^User 
attribute.  RFIs  will  wait  in  this  queue  until  a  member  of  the  UserAnalysts  set 
that  corresponds  to  the  value  of  the  User  attribute  becomes  available.  This  setup 
was  chosen  since  it  is  primarily  the  duty  of  the  user  to  determine  if  any  existing 
information  will  meet  their  needs  prior  to  requesting  new  collection.  When  an  item 
leaves  the  queue,  it  is  immediately  checked  to  see  if  it  still  meets  the  timeliness 
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requirement.  This  is  done  by  means  of  the  following  logical  expression  for  Timely 
which  evaluates  to  either  1  (true)  or  0  (false): 


{TNOW  —  Entity. CreateTime)  —  TimeR  <  Timely  .Threshold  (3.1) 

This  expression  takes  the  difference  between  the  time  an  RFI  has  been  in 
process  and  the  time  required  and  compares  that  difference  with  the  threshold  value 
which  has  a  default  value  of  0.  If  the  difference  is  less  than  the  threshold,  then  there 
is  still  time  to  work  on  a  requirement  before  it  is  due.  In  the  event  that  an  item  is  not 
timely,  then  it  will  be  sent  to  the  Evaluation  submodel  for  statistics  collection  and 
disposal.  The  resource  that  was  seized  when  the  item  left  the  queue  is  immediately 
released  before  any  simulated  time  passes.  It  is  important  to  note  here  that  this 
can  affect  some  statistics.  In  general,  discrete  statistics  that  involve  counts,  such  as 
the  number  of  times  a  resource  was  used,  will  be  affected.  However,  time-weighted 
statistics,  such  as  resource  utilization,  will  not  be  affected  since  a  zero  simulated 
time  has  elapsed.  In  spite  of  the  effect  of  this  construction  on  the  statistics,  it  was 
implemented  to  ensure  resources  were  not  unrealistically  expended  on  requests  that 
were  no  longer  timely.  The  simulation  user  should  be  aware  that  this  construction 
for  a  timeliness  check  is  replicated  throughout  the  IPM. 

If  an  RFI  passes  the  timeliness  check,  it  encounters  a  delay  based  on  the  QualR 
and  InfoSource  attributes.  As  implemented,  the  simulation  user  would  input  the 
probability  distributions  as  an  array  of  expressions  with  one  dimension  correspond¬ 
ing  to  the  levels  of  required  quality  and  the  other  dimension  corresponding  to  the 
information  sources  list.  In  general,  if  a  dependency  is  explicitly  noted,  then  it  will 
be  represented  in  the  simulation  as  an  array  of  expressions.  The  use  of  expressions 
provides  the  flexibility  to  model  times  or  other  values  with  probability  distributions. 
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Table  3.3 

RFI  attributes  that  describe  library  search  results. 

Attribute 

Description 

FoundInLibrary 

Indicates  if  item  was  (1)  or  was  not  (0)  found  in  library 

QualA 

Indicates  quality  of  item  found  in  library  (real  value  0  to  5,  5  is  best) 

AgeMetInLibrary 

Indicates  if  item  found  does  (1)  or  does  not  (0)  meet  age  requirement 

Although  untimely  items  grab  the 
resource,  that  resource  is  released 
after  a  0  time  delay,  effectively 
not  using  the  resource. 


*This  should  not  affect  time  weighted 
statistics,  but  would  affect  discrete  statistics 
such  as  numer  of  times  a  resource  was  used 


Figure  3.5  Collection  submodel 


mathematical  functions,  logical  expressions,  or  a  combination.  As  such,  additional 
dependencies  can  easily  be  added  if  necessary. 

Once  the  delay  for  an  RFI  is  complete,  the  search  results  for  that  item  are  de¬ 
termined.  A  database  or  library  is  not  actually  searched  to  determine  the  search  re¬ 
sults,  rather  the  results  are  determined  probabilistically.  Three  attributes,  Foundln- 
Library,  QualA,  and  AgeMetInLibrary  (see  Table  3.3),  are  assigned  values  that  de¬ 
scribe  the  search  results.  The  value  assigned  to  each  of  these  attributes  is  dependent 
upon  both  the  User  and  InfoSource  attributes  of  the  RFI.  These  dependencies  al¬ 
low  the  emulation  of  the  ability  of  users  to  access  various  databases  and  libraries. 
The  FoundInLibrary  and  AgeMetInLibrary  attributes  are  only  used  in  the  user  re¬ 
quirements  planning  portion  of  the  model,  whereas,  the  QualA  attribute  is  used  and 
updated  throughout  the  entire  model.  Prior  to  leaving  the  library  search  for  either 
the  Communications  or  Evaluation  submodels,  the  resource  that  was  seized  will  be 
released,  and  the  LastStation  attribute  will  be  updated  to  a  value  that  corresponds 
to  the  library  portion  of  the  Planning  and  Direction  submodel. 
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3.3.2  Collection  Submodel 


The  Collection  submodel  corresponds  directly  to  the  collection  phase  of  the  In¬ 
telligence  Cycle.  This  purpose  of  this  submodel  is  to  task  collection  from  information 
sources  and  determine  the  quality  of  information  collected.  The  visual  structure  of 
the  Collection  submodel  is  quite  similar  to  that  of  the  library  portion  of  the  Planning 
and  Direction  submodel,  yet  the  underlying  information  differs.  Entities  arrive  from 
the  Communications  submodel  and  are  immediately  placed  in  an  LVF  queue  based 
on  the  Priority. Clobal  attribute.  Entities  wait  in  the  queue  until  an  appropriate 
resource  becomes  available.  When  a  member  of  the  Sources  resource  set  correspond¬ 
ing  to  a  value  of  the  InfoSource  attribute  becomes  available,  the  RFI  in  the  queue 
that  has  the  highest  global  priority  (i.e.,  lowest  value  of  Priority .Clobal)  for  a  given 
InfoSource  value  will  be  removed  from  the  queue.  When  an  entity  leaves  the  queue, 
it  will  immediately  undergo  a  timeliness  check  using  Equation  (3.1).  If  the  item 
has  failed  to  meet  the  timeliness  requirement,  then  the  resource  that  was  seized  will 
be  released  before  any  simulated  time  passes.  Again,  it  is  important  that  the  sim¬ 
ulation  user  understands  that  this  will  affect  discrete  count-type  statistics  but  not 
time-weighted  statistics  involving  the  resources.  The  LastStation  attribute  will  then 
be  assigned  a  value  corresponding  to  the  Collection  submodel,  and  the  RFI  will 
be  sent  to  the  Evaluation  submodel  for  statistics  collection.  On  the  other  hand,  if 
the  item  is  still  timely,  then  the  RFI  will  undergo  a  delay  based  on  the  QualR  and 
InfoSource  attributes.  The  delay  corresponds  to  the  time  taken  to  collect  informa¬ 
tion  and  is  determined  from  an  array  of  expressions  (e.g.,  probability  distributions, 
mathematical  functions,  or  logical  expressions)  that  is  indexed  on  the  value  of  those 
attributes.  Once  the  delay  is  complete,  a  value  rating  the  actual  quality  of  collected 
information  is  assigned  to  the  QualA  attribute.  The  actual  quality  is  determined 
from  an  array  of  expressions  based  on  the  QualR  and  InfoSource  attributes.  Prior 
to  leaving  the  Collection  submodel,  the  seized  resource  will  be  released  and  the 
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Although  unti  mely  items  grab  the 
resource,  that  resource  is  released 
after  a  0  time  delay,  effectively 
not  using  the  resource. 


’This  should  not  affect  time  weighted 
statistics,  but  would  affect  discrete  statistics 
such  as  numer  of  times  a  resource  was  used. 

Figure  3.6  Processing  submodel 

LastStation  attribute  will  then  be  assigned  a  value  corresponding  to  the  Collection 
submodel.  Finally,  the  RFI  will  be  sent  back  to  the  Communications  submodel. 

3.3.3  Processing  and  Exploitation  Submodel 

The  Processing  and  Exploitation  submodel  corresponds  directly  to  the  Process¬ 
ing  and  Exploitation  phase  of  the  Intelligence  Cycle.  In  order  to  further  modularize 
the  model,  this  submodel  has  been  broken  into  two  portions,  one  for  processing  and 
one  for  exploitation.  Other  than  being  contained  in  the  same  submodel,  they  are 
independent  of  each  other.  This  independence  allows  entities  to  go  through  either 
portion  alone  or  both. 

3.3.3. 1  Processing 

The  purpose  of  the  processing  portion  of  the  Processing  and  Exploitation  submodel 
is  to  task  processing  resources  and  determine  the  effects  of  processing  on  the  qual¬ 
ity  of  information.  As  entities  enter  processing,  they  are  placed  in  an  LVF  queue 
based  on  the  Priority.Clobal  attribute.  The  PEIs  will  remain  in  the  queue  until  a 
member  of  the  InfoProcessors  resource  set  corresponding  to  the  value  stored  in  the 
InfoSource  attribute.  Once  an  item  is  taken  out  of  the  queue,  a  timeliness  check  is 


Release 

Leave 

InfoProcessers 

' — HProcessingStatiorf— ^ 

Processing  to 

n  Variable  1  J 

Evaluation 
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done  using  Equation  (3.1)  to  avoid  wasting  resources  on  RFIs  that  are  not  timely. 
If  an  RFI  fails  the  timeliness  check,  then  the  seized  resource  is  released  before  any 
simulated  time  passes.  As  noted  before,  this  will  affect  discrete  count-type  statistics 
but  not  time-weighted  statistics  involving  the  resources.  Just  prior  to  leaving  for  the 
Evaluation  submodel,  the  LastStation  attribute  is  assigned  a  value  corresponding  to 
the  processing  portion  of  the  model.  If  an  RFI  passes  the  timeliness  check,  it  will 
undergo  a  delay  corresponding  to  the  time  needed  to  process  the  data  associated 
with  that  RFI.  This  delay  is  drawn  from  an  array  of  expressions  that  is  dependent 
on  QualARank  and  the  InfoSource  attribute. 

QualARank  is  the  following  expression  that  ranks  the  value  of  the  QualA  at¬ 
tribute  on  a  scale  from  one  to  six: 

QualARank  =  MN  {MX  {ANINT  {QualA) ,  0)  +  1,  6)  (3.2) 

The  values  are  adjusted  to  range  from  one  to  six  rather  than  zero  to  hve  since 
the  hrst  element  of  arrays  in  Arena  have  an  index  value  of  one.  To  clarify  some  of 
Equation  (3.2),  the  MN()  function  takes  the  minimum  of  a  list  of  values,  the  MX() 
function  takes  the  maximum  of  a  list  of  values,  and  the  ANINT ()  function  rounds 
a  value  to  the  nearest  integer.  In  some  studies  it  may  be  more  appropriate  for  the 
value  of  QualA  to  be  truncated  instead  of  rounded  for  this  ranking.  If  this  is  the 
case,  then  the  simulation  user  could  simply  replace  the  ANINT ()  function  with  the 
AINT()  function  which  truncates  to  an  integer. 

When  the  delay  for  processing  is  complete,  the  QualA  attribute  is  updated. 
This  update  allows  for  the  effect  of  processing  on  actual  information  quality  to  be 
modeled.  The  update  to  QualA  here  is  only  dependent  on  the  InfoSource  attribute. 
The  next  four  blocks  in  the  model  add  rudimentary  support  for  information  fusion. 

Some  proportion  of  RFIs  will  be  selected  at  random  to  undergo  fusion.  For 
those  that  undergo  fusion,  another  timeliness  check  is  conducted  using  Equation 
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such  as  numer  of  times  a  resource  was  used. 

Figure  3.7  Exploitation  submodel 

(3.1).  If  the  RFI  passes  the  timeliness  check,  then  it  will  undergo  a  delay  that  is 
dependent  on  QualARank  in  Equation  (3.2)  and  the  InfoSource  attribute.  Once  the 
delay  is  complete,  an  additional  update  is  made  to  QualA  that  is  dependent  on  the 
InfoSource  attribute.  This  update  to  QualA  is  done  to  reflect  how  fusion  during 
processing  affects  actual  quality.  The  information  fusion  modeled  here  amounts  to 
additional  processing  on  a  given  RFI.  The  assumption  behind  this  implementation 
is  that  all  other  information  required  for  fusion  is  available.  This  assumption  was 
made  since  there  is  no  association  between  individual  RFIs  in  the  current  model. 
Developing  a  more  robust  fusion  model  would  require  RFIs  to  have  multiple  data 
sources  which  is  beyond  the  scope  of  this  study.  As  such,  the  fusion  portion  of  this 
model  has  limited  applicability. 

Prior  to  leaving  this  portion  of  the  model,  the  resources  seized  by  the  RFI 
are  released.  The  LastStation  attribute  is  assigned  a  value  that  corresponds  to  the 
processing  portion  of  the  model,  and  the  RFI  is  sent  back  to  the  Communications 
submodel. 
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3. 3. 3. 2  Exploitation 


The  exploitation  portion  of  the  Processing  and  Exploitation  submodel  is  used  to  task 
information  specialists  to  perform  exploitation  of  information  and  determine  the  ef¬ 
fect  of  exploitation  on  information  quality.  This  portion  of  the  model  visually  looks 
the  same  as  the  processing  portion  of  the  model,  but  the  underlying  information 
differs.  As  RFIs  arrive  from  the  Communications  submodel  they  are  placed  in  an 
LVF  queue  based  on  Priority. Global.  When  a  member  of  the  Info  Specialists  resource 
set  corresponding  to  the  InfoSource  attribute  of  an  RFI  in  the  queue  becomes  avail¬ 
able,  the  RFI  will  undergo  a  timeliness  check  using  Equation  (3.1).  If  the  timeliness 
check  fails,  then  the  seized  resource  will  be  released  and  no  simulation  time  will 
have  elapsed.  As  before,  this  construction  affects  discrete  count  statistics  but  not 
time-weighted  statistics.  Prior  to  leaving  for  the  Evaluation  submodel,  the  LastSta- 
tion  attribute  is  assigned  a  value  corresponding  to  the  exploitation  portion  of  the 
model.  If  the  timeliness  check  is  passed,  then  the  RFI  will  be  delayed  according  to 
the  time  needed  to  exploit  information.  As  implemented,  the  delay  for  exploitation 
is  dependent  upon  QualR,  QualARank  (Equation  (3.2)),  and  InfoSource.  Since  this 
is  technically  a  three-dimensional  array  of  expressions,  it  could  not  be  implemented 
as  a  single  array  in  Arena.  In  order  to  overcome  this,  six  expression  arrays  were  cre¬ 
ated.  The  hrst  array  was  indexed  on  QualR  and  drew  values  from  Eve  other  arrays 
(one  for  each  possible  value  of  QualR)  indexed  on  QualARank  and  InfoSource.  Once 
the  exploitation  is  complete,  the  affect  on  actual  quality  will  be  assessed.  The  value 
of  QualA  is  assigned  a  value  from  an  expression  array  indexed  on  the  InfoSource 
attribute. 

As  with  the  processing  portion  of  the  model,  this  portion  contains  a  rudimen¬ 
tary  fusion  model.  This  fusion  model  makes  the  assumption  that  all  other  required 
information  is  available  at  the  time.  Consequently,  fusion  as  implemented  has  lim¬ 
ited  applicability.  For  the  proportion  of  RFIs  that  are  randomly  selected  to  undergo 
fusion,  a  timeliness  check  will  be  conducted.  If  an  item  passes  the  timeliness  check. 
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a  delay  will  be  incurred  that  is  based  on  the  QualARank  expression  (Equation  (3.2)) 
and  InfoSource  attribute.  The  effect  of  fusion  during  exploitation  on  actual  qual¬ 
ity  is  then  assessed  when  the  QualA  attribute  is  updated  based  on  the  InfoSource 
attribute. 

Prior  to  leaving  the  exploitation  portion  of  the  model  for  the  Communications 
submodel,  the  resources  used  for  exploitation  are  released.  Additionally,  the  Last- 
Station  attribute  is  assigned  a  value  corresponding  to  the  exploitation  portion  of  the 
model. 

3.3.4  Analysis  and  Production  Submodel 

The  Analysis  and  Production  submodel  corresponds  to  the  Analysis  and  Pro¬ 
duction  phase  of  the  Intelligence  Cycle.  As  with  the  Processing  and  Exploitation 
submodel,  this  submodel  consists  of  two  independent  portions.  As  the  name  of  the 
submodel  suggests,  one  portion  is  for  analysis  and  the  other  for  production.  For 
some  information  sources,  it  may  be  the  case  that  analysis  and  production  cannot 
be  distinguished.  If  so,  then  the  simulation  user  could  set  up  the  simulation  to  use 
one  of  them  and  not  the  other  and  assign  appropriate  delay  and  quality  update  ex¬ 
pressions  to  the  part  that  is  used.  To  accomplish  this  an  expression  more  complex 
than  a  simple  proportion  will  be  needed  when  the  Analyze  and  Produce  attributes 
are  assigned  (see  Section  3.3. 1.1). 

3. 3. 4-1  Analysis 

The  analysis  portion  of  the  model  is  visually  similar  to  the  processing  and  exploita¬ 
tion  portions  of  the  model  but  the  underlying  information  differs.  When  PFIs  enter 
the  analysis  portion  of  the  model  they  enter  an  LVF  queue  based  on  Priority^Global. 
When  a  member  of  the  AllSourceAnalysts  resource  set  becomes  available,  the  RFI 
with  the  highest  priority  is  removed  from  the  queue.  If  more  than  one  member  of 
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Figure  3.8  Analysis  submodel 


the  AllSourceAnalysts  set  is  available,  then  the  one  chosen  is  selected  at  random. 
The  AllSourceAnalysts  set  is  divided  into  thirteen  specialties  that  may  or  may  not 
correspond  to  the  Sources  resource  set,  depending  on  the  study.  The  specialty  of 
the  analyst  used  is  stored  in  the  AnalystSpecialty  attribute.  Upon  exiting  the  queue, 
a  timeliness  check  is  done  using  Equation  (3.1).  If  an  item  is  not  timely,  then  the 
resource  is  released  prior  to  any  simulation  time  elapsing.  As  noted  before,  this 
affects  count  statistics  but  not  time-weighted  statistics.  The  LastStation  attribute 
is  assigned  a  value  corresponding  to  the  analysis  portion  of  the  model  prior  to  be¬ 
ing  sent  to  the  Evaluation  submodel  for  statistics  collection.  The  RFIs  that  are 
still  timely  incur  a  delay  based  on  QualR,  QualARank  (Equation  (3.2)),  and  In- 
foSource.  This  three-dimensional  expression  array  is  implemented  similarly  to  the 
three-dimensional  array  discussed  in  Section  3. 3. 3. 2.  After  the  delay,  the  effect  of 
analysis  on  actual  information  quality  is  assessed  by  updating  the  QualA  attribute 
from  an  expression  array  based  on  the  InfoSource  and  AnalystSpecialty  attributes. 
As  with  the  processing  and  exploitation  portions  of  the  model,  the  analysis  por¬ 
tion  contains  a  rudimentary  capacity  for  fusion.  It  contains  the  same  assumption 
that  any  additional  required  information  is  available  for  fusion.  For  the  proportion 
of  RFIs  that  undergo  fusion,  a  timeliness  check  is  done  using  Equation  (3.1).  For 
items  that  are  timely,  they  will  undergo  a  delay  based  on  the  QualARank  expression 
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such  as  numer  of  times  a  resource  was  used. 

Figure  3.9  Production  submodel 

(Equation  (3.2))  and  InfoSource  attribute.  Upon  completion  of  fusion,  the  QualA 
attribute  is  updated  based  on  the  InfoSource  attribute  to  reflect  how  fusion  during 
analysis  affects  actual  quality.  Prior  to  RFIs  leaving  this  portion  of  the  model  for 
the  Communications  submodel,  the  resources  used  are  released  and  the  LastStation 
attribute  is  assigned  a  value  corresponding  to  the  analysis  portion  of  the  model. 

3. 3. 4- 2  Production 

The  production  portion  of  the  model  visually  resembles  the  Collection  submodel 
rather  than  the  analysis  portion  of  the  model  since  it  does  not  contain  any  blocks  to 
model  fusion.  As  RFIs  arrive  from  the  Communications  submodel,  they  are  placed 
in  an  LVF  queue  based  on  Priority ^Clobal.  A  member  of  the  AllSourceAnalysts  re¬ 
source  set  is  selected  at  random  if  more  than  one  is  available.  The  specialty  of  the 
selected  analyst  is  stored  in  the  AnalystSpecialty  attribute  of  the  RFI.  Note  that 
if  the  RFI  has  already  undergone  analysis,  this  value  will  be  overwritten  and  the 
selected  specialty  is  independent  of  the  specialty  used  for  analysis.  If  the  simula¬ 
tion  user  needs  the  specialty  to  be  identical,  then  they  could  modify  the  seize  block 
to  select  a  specihc  member  of  the  AllSourceAnalysts  set  based  on  the  value  stored 
in  the  AnalystSpecialty  attribute.  Care  must  be  taken  to  account  for  those  items 
that  have  not  undergone  analysis  since  the  AnalystSpecialty  attribute  will  contain 
the  invalid  set  index  value  of  zero.  When  an  RFI  leaves  the  queue,  it  undergoes  a 
timeliness  check  using  Equation  (3.1).  For  those  items  that  fail  the  timeliness  check, 
the  resource  will  be  released  after  no  simulation  time  has  elapsed.  As  noted  before. 
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this  affects  discrete  count  statistics  but  not  time-weighted  statistics.  Additionally, 
the  LastStation  attribute  will  be  assigned  a  value  corresponding  to  the  production 
portion  of  the  model  prior  to  being  sent  to  the  Evaluation  submodel.  Those  items 
that  pass  the  timeliness  check  will  incur  a  delay  for  production  based  on  QualR,  , 
QualARank  (Equation  (3.2)),  and  InfoSource.  This  three-dimensional  array  is  imple¬ 
mented  similarly  to  the  three-dimensional  array  discussed  in  Section  3. 3. 3. 2.  Once 
the  production  delay  is  complete,  the  effect  of  production  on  quality  is  determined 
by  updating  QualA  from  an  expression  array  depending  on  the  InfoSource,  and  An- 
alystSpecialty  attributes.  After  the  quality  is  updated,  the  resource  being  used  is 
released  and  the  LastStation  attribnte  is  assigned  a  value  corresponding  to  the  pro¬ 
duction  portion  of  the  model.  Finally,  the  RFI  is  sent  back  to  the  Communications 
submodel. 

3.3.5  Dissemination  and  Integration  Submodel 

The  Dissemination  and  Integration  submodel  corresponds  to  the  Dissemination 
and  Integration  phase  of  the  Intelligence  Cycle.  This  portion  of  the  model  is  intended 
to  reflect  more  of  the  user  process  and  as  snch  involves  the  use  of  user  resources. 
Similar  to  other  submodels,  this  submodel  is  divided  into  two  independent  portions, 
one  for  dissemination  and  one  for  integration.  Integration  here  relates  to  the  user 
integrating  information  into  their  processes  not  information  integration  as  related  to 
exploitation  or  analysis. 

3.3.5. 1  Dissemination 

The  purpose  of  the  dissemination  portion  of  the  model  is  to  reflect  the  user  process 
of  actually  acquiring  information  that  has  been  through  some  portion  of  the  intelli¬ 
gence  process.  RFIs  arrive  from  the  Communications  submodel  and  are  placed  in 
an  LVF  quene  based  on  Priority-User.  Note  that  this  differs  from  most  of  the  qnenes 
to  this  point,  and  is  based  off  of  user  priorities  rather  than  global  priorities  since  it  is 
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Figure  3.10  Dissemination  submodel 
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Figure  3.11  Integration  submodel 


primarily  a  user  process.  RFIs  wait  in  the  queue  until  a  member  of  the  User  Analysts 
resource  set  that  corresponds  to  the  value  of  the  User  attribute  becomes  available. 
Note  that  no  communications  resources  are  currently  used  here.  When  an  item  is  re¬ 
moved  from  the  queue,  a  timeliness  check  is  performed  using  Equation  (3.1).  Those 
items  that  are  not  timely  are  sent  to  the  Evaluation  submodel  after  releasing  the 
seized  resource  and  assigning  LastStation  a  value  corresponding  to  the  dissemination 
portion  of  the  model.  The  resource  is  released  after  zero  simulation  time  has  passed 
which,  as  noted  before,  affects  discrete  count  statistics  but  not  time-weighted  statis¬ 
tics.  Those  items  that  are  timely  will  then  undergo  a  delay  that  depends  on  the  User 
attribute.  When  dissemination  is  complete,  the  effect  on  quality  is  determined  by 
updating  the  QualA  attribute  from  an  expression  array  based  on  InfoSource.  The 
seized  resource  is  then  released,  and  the  LastStation  attribute  is  assigned  a  value 
corresponding  to  the  dissemination  portion  of  the  model. 
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3. 3. 5. 2  Integration 


The  purpose  of  the  integration  portion  of  the  model  is  to  reflect  the  integration  of 
information  into  user  processes.  The  use  or  non-use  of  this  portion  of  the  model 
changes  the  interpretation  of  the  timeliness  measure.  If  this  portion  of  the  model  is 
not  used,  then  timeliness  is  essentially  determined  by  when  a  user  receives  requested 
information.  If  this  portion  of  the  model  is  used,  then  timeliness  is  determined  by 
when  a  user  is  able  to  use  information  they  have  received.  As  RFIs  arrive  from  the 
Communications  submodel,  they  enter  an  LVF  queue  based  on  the  Priority ^User 
attribute.  They  will  wait  in  the  queue  until  a  member  of  the  UserAnalysts  resource 
set  corresponding  to  the  User  attribute  becomes  available.  Upon  leaving  the  queue, 
the  RFIs  undergo  a  timeliness  check  using  Equation  (3.1).  Those  items  that  fail  the 
timeliness  check  will  release  the  resource  before  any  simulation  time  elapses.  As  noted 
before,  this  affects  count  statistics  but  not  time-weighted  statistics.  The  LastStation 
attribute  will  be  assigned  a  value  corresponding  to  the  integration  portion  of  the 
model  prior  to  sending  the  untimely  RFIs  to  the  Evaluation  submodel.  Those  items 
that  pass  the  timeliness  check  will  incur  a  delay  based  on  the  User  attribute.  Upon 
completion  of  integration  the  affect  on  quality  is  assessed  by  updating  the  QualA 
attribute  from  an  expression  array  based  on  the  Info  Source  attribute.  Finally,  the 
seized  resource  is  released,  the  LastStation  attribute  is  assigned  a  value  for  the  in¬ 
tegration  portion  of  the  model,  and  the  RFI  is  sent  back  to  the  Communications 
submodel. 

3.3.6  Evaluation  Submodel 

The  purpose  of  the  Evaluation  submodel  is  to  collect  statistics  and  dispose  of 
entities.  If  there  are  no  problems  with  the  model,  all  entities  should  exit  the  system 
through  the  Evaluation  submodel.  In  addition  to  the  default  statistics  gathered, 
additional  statistics  are  needed  to  evaluate  the  quality,  timeliness,  and  information 
needs  satisfaction  (as  it  relates  to  meeting  both  quality  and  timeliness)  measures. 
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Table  3.4  Evaluation  submodel  expressions  assigning  variables  for  statistics. 


S_Nuni_Timely_Priority(Priority_User) 

S_Num_Timely_User(User) 

S_Num_Timely_Standarcl(Standard+1) 

S_Num_Qual_Priority(Priority_User) 

S_Num_Qual_User(User) 

S_Num_Qual_Standard(Standard+1) 

S_Num_QT_Priority(Priority_User) 

S_Num_QT_User(User) 

S_Num_QT_Standard(Standard+1) 

S_Nuni_Out_Priority(Priority_User) 

S_Num_Out_User(User) 

S_Num_0ut_Standard(Standard+1) 

S_Num_Timely_Src(lnfoSource) 

S_Nuni_Qual_Src(lnfoSource) 

S_Num_QT_Src(lnfoSource) 

S_Num_Out_Src(lnfoSource) 

S_Tot_WaitTime_Priority(Priority_User) 

S_Max_WaitTime_Priority(Priority_User) 


Timely‘(S_Num_Timely_Priority(Priority_User)+1)  +  (1-Timely)*S_Num_Timely_Priority(Priority_User) 
Timely‘(S_Num_Timely_User(User)+1)  +  (1-Timely)*S_Num_Timely_User(User) 
Timely*(S_Num_Timely_Standard(Standard+1)+1)  +  (1-Timely)*S_Num_Timely_Standard(Standard+1) 
QualMet*(S_Num_Qual_Priority(Priority_User)+1)  +  (1-QualMet)*S_Num_Qual_Priority(Priority_User) 
QualMet*(S_Num_Qual_User(User)+1)  +  (1-QualMet)‘S_Num_Qual_User(User) 
QualMet*(S_Num_Qual_Standard(Standard+1)+1)  +  (1-QualMet)*S_Num_Qual_Standard(Standard+1) 
Timely*QualMet*(S_Num_QT_Priority(Priority_User)+1)  +  (1-Timely*QualMet)*S_Num_QT_Priority(Priority_User) 
Timely‘QualMet‘(S_Num_QT_User(User)+1)  +  (1-Timely*QualMet)‘S_Num_QT_User(User) 
Timely*QualMet*(S_Num_QT_Standard(Standard+1)+1)  +  (1-Timely*QualMet)‘S_Nurri_QT_Standard(Standard+1) 
S_Num_Out_Priority(Priority_User)+1 
S_NurTi_0ut_User(User)+1 
S_Num_0ut_Standard(Standard+1)+1 

Timely‘(S_Num_Timely_Src(lnft)Source)+1)  +  (1-Timely)‘S_Num_Timely_Src(lnfoSource) 
QualMet*(S_Num_Qual_Src(lnfoSource)+1)  +  (1-QualMet)*S_Num_Qual_Src(lnfoSource) 
Timely*QualMet‘(S_Num_QT_Src(lnfoSource)+1)  +  (1-Timely‘QualMet)*S_Num_QT_Src(lnfoSource) 
S_Num_Out_Src(lnfoSource)+1 
S_Tot_WaitTime_Priority(Priority_User)  +  Entity  .WaitTime 
MX(S_Max_WaitTime_Priority(Priority_User),  Entity  .WaitTime) 


The  additional  data  needed  is  gathered  as  entities  enter  the  Evaluation  submodel. 
The  data  is  stored  in  variables  and  is  gathered  according  to  the  expressions  given  in 
Table  3.4. 

The  expressions  make  extensive  use  of  the  Timely  and  QualMet  logical  expres¬ 
sions  which  evaluate  to  0  or  1  to  track  many  of  the  counts  stored  in  the  variables. 
The  QualMet  expression  is  given  in  the  following  equation: 


QualR  —  QualA  <  QualM et ^Threshold  (3.3) 


This  expression  compares  the  difference  between  the  required  and  actual  qual¬ 
ity  with  a  simulation  user  specihed  threshold  value  which  by  default  is  zero.  At  the 
end  of  a  simulation  run,  these  variables  are  used  to  generate  various  statistics. 

Another  feature  that  is  partially  implemented  in  the  Evaluation  submodel  is 
an  initial  capability  to  allow  feedback  of  REIs  into  the  remainder  of  the  system. 
However,  this  capability  should  not  be  used  since  the  model  is  not  programmed  to 
do  anything  useful  with  feedback  items.  As  currently  implemented,  any  entity  that 
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leaves  the  Evaluation  submodel  will  travel  through  the  Communications  submodel 
to  the  planning  portion  of  the  model  where  it  will  be  immediately  disposed. 

3.3.7  Communications  Submodel 

The  primary  purpose  of  the  Communications  submodel  is  to  route  entities 
between  the  other  submodels  as  well  as  provide  appropriate  communications  delays 
and  resources.  The  communications  model  is  broken  into  two  portions,  the  first 
receives  entities  from  the  other  parts  of  the  model  and  the  second  portion  sends 
entities  to  other  portions  of  the  model.  The  primary  tasks  of  the  hrst  portion  of  the 
model  are  to  determine  where,  i.e.,  which  portion  of  the  model,  the  entity  should 
go  to  next,  task  appropriate  resources  for  communications,  and  send  items  to  the 
second  portion  of  the  model.  The  primary  tasks  of  the  second  portion  of  the  model 
are  to  update  information  quality  and  send  the  RFI  to  the  submodel  determined 
by  the  hrst  portion  of  the  Communications  submodel.  Prior  to  discussing  each  of 
the  paths,  it  is  important  to  note  that  only  minimal  communications  delays  and 
resources  are  modeled.  A  detailed  communications  model  is  not  appropriate  due  to 
the  high  level  of  the  overall  model. 

The  hrst  portion  of  the  Communications  submodel,  shown  in  Figure  3.12,  has 
10  entry  points  corresponding  to  the  various  portions  of  the  model  discussed  above. 
From  top  to  bottom,  as  entities  arrive  from  the  planning  portion  of  the  model,  a 
check  is  done  to  see  if  the  NextStation  attribute  is  set  to  a  value  other  than  zero. 
This  occurs  when  items  need  to  undergo  a  library  search,  and  it  is  set  in  the  planning 
portion  of  the  model.  If  the  value  is  greater  than  zero,  then  the  entities  are  routed 
to  the  second  portion  of  the  Communications  submodel.  Otherwise,  the  NextStation 
attribute  is  assigned  a  value  using  VBA  block  1.  The  VBA  blocks  were  used  in 
lieu  of  Assign  blocks  since  the  required  logical  expressions  exceeded  the  maximum 
length  of  an  expression  field.  Furthermore,  the  logic  of  the  code  in  VBA  is  easier 
to  create  and  understand  than  equivalent  but  exceptionally  long  logical  expressions. 
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’  VBA  blocks  instead  of  Assign  blocks 
since  the  expressions  are  too  long  and 
complex  to  fit  in  the  expression  field 


Enter  Comms 
from 

Exploitation 


4)  Assign  NextStation 

after  Exploitation 


Route  to  Exit  5 


Enter  Comms 

5)  Assion  NextStation 

from  Analysis 

after  Analysis  ^  ' 

Enter  Comms 
from 

Production 


6)  Assign  NextStation 


after  Production 


Route  to  Exit  7 


Enter  Comms 
from 

Dissemination 


Enter  Comms 
from  Integration 


Figure  3.12 


First  portion  of  Communications  submodel 
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Leave  to 
Planning 


NextStation—PlanningSiatlon 
N  exist  aii  0  n==  Li  b  re  lyS  te  ti  o  n 
NextStaiion—CollectioriStetion 
N  exist  at  i  0  n==  P  ro  cessi  n  g  Stat  i  o  n 
NextStatiorr^ExploitationStation 
N  extStat  i  onss  An  a  lysisStati  o  n 
NextStation==ProduetionStation 
NextStation—DisseminationStation 
N  extStat  i  I  nte  g  rati  o  n  Stall  on 

N  extStat  i  0  n==  E  va  lu  ati  o  n  Stati  on 


Leave  to 
Library 


Leave  to 

Collection 

Leave  to 
Processing 


Leave  to 
Exploitation 


Leave  to 
Analysis 


LastStation  and  NexlStation 
Attribute  identifiers: 

(Use  variables  instead  of  numbers) 

1)  Planning 

2)  Library 

3)  Collection 

4)  Processing 

5)  Exploitation 

6)  Analysis 

7)  Production 

8)  Dissemination 

9)  Integration 

1 0)  Evaluation 


Leave  to 
Production 


Bad  Path  Dispose 


Leave  to 
Dissemination 


Leave  to 
Integration 


Leave  to 
Evaluation 


Figure  3.13 


Second  portion  of  Communications  submodel 
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Algorithm  1  Portion  of  code  for  VBA  block  1. 

’  Determine  NextStation  after  leaving  Planning 
If  Need_Collect  =  1  Then 

Next_Station  =  Stn_Collection 
Elself  Need_Process  =  1  Then 

Next_Station  =  Stn_Processing 
Elself  Need_Exploit  =  1  Then 

Next_Station  =  Stn_Exploitation 
Elself  Need_Analyze  =  1  Then 

Next_Station  =  Stn_Analysis 
Elself  Need_Produce  =  1  Then 

Next_Station  =  Stn_Production 
Elself  Need_Disseminate  =  1  Then 

Next_Station  =  Stn_Dissemination 
Elself  Need_lntegrate  =  1  Then 

Next_Station  =  Stn_lntegration 

Else 

Next_Station  =  Stn_Evaluation 
End  If 


The  portion  of  the  code  that  contains  the  logic  for  where  to  go  next  is  given  in 
Algorithm  1.  The  complete  code  for  all  VBA  blocks  is  given  in  Appendix  B.  The 
code  sequentially  examines  the  values  of  the  attributes  given  in  Table  3.2  and  sets 
the  NextStation  attribute  to  the  value  corresponding  to  the  hrst  attribute  that  has  a 
value  of  one.  This  is  similar  to  the  VBA  blocks  used  for  other  entry  points  into  the 
Communications  submodel.  The  second  entry  block  receives  entities  from  the  library 
portion  of  the  model.  As  the  RFIs  arrive,  the  NextStation  attribute  is  assigned  a 
value  corresponding  to  the  planning  portion  of  the  model.  The  reason  for  this  is  that 
the  planning  portion  of  the  model  needs  to  determine  what  to  do  with  the  search 
results.  For  more  information,  see  Section  3.3.1.  RFIs  are  then  sent  to  the  second 
portion  of  the  Communications  submodel. 

The  third  entry  block  receives  RFIs  from  the  Collection  submodel  and  assigns 
the  NextStation  attribute  according  to  VBA  block  2.  The  logic  is  similar  to  that 
depicted  in  Algorithm  1  except  that  the  value  of  the  Collect  attribute  would  not 
be  checked  since  the  RFI  has  already  been  through  collection.  The  RFIs  then 
enter  a  FIFO  queue  and  wait  until  a  member  of  the  Comms Collection  resource  set 
corresponding  to  the  InfoSource  attribute  is  available.  Upon  leaving  the  queue,  a 
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delay  is  incurred  based  on  the  InfoSource  attribute.  Once  the  delay  is  complete, 
the  resource  is  released  and  the  value  of  the  PostInfo  variable  is  checked.  If  the 
value  is  0,  then  the  RFI  will  be  routed  to  the  second  portion  of  the  Communications 
submodel.  If  the  value  is  1,  then  the  RFI  will  enter  an  LVF  queue  for  posting 
based  on  the  Priority  ^Global  attribute.  The  RFI  will  wait  in  the  queue  until  a 
CommsPost  resource  becomes  available.  Note  that  this  resource  is  shared  with  two 
other  posting  queues  which  have  equal  priority  for  using  resources.  Upon  leaving 
the  queue,  the  entity  will  undergo  a  delay  based  on  the  InfoSource  attribute  and 
release  the  resource  when  the  delay  is  complete.  The  RFI  will  then  be  routed  to 
the  second  portion  of  the  Communications  submodel.  It  is  important  to  note  here 
that  this  communications  modeling  is  not  robust  and  does  not  adequately  capture 
the  complexity  of  the  communications  environment.  As  such  it  becomes  a  prime 
candidate  for  further  study. 

The  fourth  entry  block  receives  RFIs  from  the  processing  portion  of  the  model. 
As  the  entities  arrive,  the  LastStation  attribute  is  assigned  a  value  by  VBA  block  3. 
The  logic  is  similar  to  that  depicted  in  Algorithm  1  except  that  it  begins  with  a  check 
for  needing  exploitation  and  does  not  check  if  prior  steps  are  needed.  The  value  of 
the  PostInfo  variable  is  then  checked  and  routed  according  to  its  value.  If  the  value 
is  0,  then  the  RFI  is  sent  to  the  second  portion  of  the  Communications  submodel. 
If  the  value  is  1,  then  the  RFI  will  enter  an  LVF  queue  based  on  the  Priority  ^Global 
attribute.  It  will  wait  until  a  CommsPost  resource  becomes  available  and  undergo 
a  delay  based  on  the  InfoSource  attribute.  Note  that  the  CommsPost  resource  is 
shared  with  two  other  posting  queues.  Once  the  delay  is  complete,  the  RFI  is  sent 
to  the  second  portion  of  the  Communications  submodel. 

The  fifth  entry  block  receives  RFIs  from  the  exploitation  portion  of  the  model 
and  assigns  the  LastStation  attribute  a  value  with  VBA  block  4.  The  code  for  this 
VBA  block  is  similar  to  Algorithm  1  except  that  it  begins  with  a  check  for  needing 
analysis  and  does  not  check  if  prior  steps  are  needed.  Depending  on  the  value  of  the 
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PostInfo  variable,  the  entity  will  either  be  sent  directly  to  the  second  portion  of  the 
Communications  submodel  or  will  enter  an  LVF  queue  based  on  the  Priority. Global 
attribute.  If  the  value  is  1,  then  the  entity  will  enter  the  queue  and  wait  until  a 
CommsPost  resource  becomes  available.  When  the  RFI  leaves  the  queue,  it  will 
undergo  a  delay  based  on  the  InfoSource  attribute.  Once  the  delay  is  complete,  the 
RFI  is  sent  to  the  second  portion  of  the  Communications  submodel. 

The  next  hve  entry  blocks  receive  RFIs  from  the  analysis,  production,  dissemi¬ 
nation,  integration,  and  evaluation  portions  of  the  model  respectively.  No  delays  are 
incurred  as  entities  go  through  these  parts  of  the  model.  The  LastStation  attribute 
is  assigned  a  value  in  each  case  by  examining  the  attributes  for  later  portions  of 
the  model  similar  to  Algorithm  1.  The  assign  blocks  contain  the  logical  expression 
equivalent  of  the  VBA  code  and  are  used  since  the  expressions  are  relatively  short. 
Upon  arriving  from  integration  the  only  option  is  to  be  sent  to  the  Evaluation  sub¬ 
model.  The  entities  arriving  from  the  Evaluation  submodel  are  sent  to  the  planning 
portion  of  the  model,  however  as  noted  in  the  previous  section  the  planning  portion 
of  the  model  will  simply  dispose  of  the  items.  For  all  hve  paths,  the  entities  will  be 
sent  to  the  second  portion  of  the  Communications  submodel. 

The  second  portion  of  the  Communications  submodel,  shown  in  Figure  3.13, 
receives  REIs  from  the  hrst  portion  of  the  submodel.  The  hrst  thing  to  occur  is  an 
update  to  the  QualA  attribute  to  rehect  the  effect  of  communications  on  the  quality 
of  information.  The  update  here  is  only  dependent  upon  the  InfoSource  attribute 
and  occurs  every  time  an  entity  goes  through  the  submodel.  Depending  on  the 
simulation  setup,  some  entities  may  travel  through  this  quality  update  as  many  as 
10  times.  This  may  be  an  item  to  check  if  output  quality  is  less  than  expected. 
Once  the  quality  has  been  updated,  REIs  undergo  a  timeliness  check  according 
to  Equation  (3.1).  Those  items  that  are  no  longer  timely  are  sent  immediately 
to  the  Evaluation  submodel  for  statistics  collection  and  disposal.  Otherwise,  the 
value  of  the  NextStation  attribute  is  used  to  branch  to  the  various  portions  of  the 
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model.  If  the  NextStation  attribute  contains  an  unknown  value,  entities  will  be 
immediately  disposed.  This  was  implemented  for  model  verihcation  and  no  entities 
should  be  disposed  of  here.  The  use  of  variables  containing  known  values  helps  to 
avoid  incorrect  routing. 

3.4  Implementation  of  Measures 

The  primary  measures  of  interest,  quality,  quantity,  timeliness,  and  informa¬ 
tion  needs  satisfaction  (as  it  relates  to  meeting  both  quality  and  timeliness)  are 
implemented  for  a  variety  of  aspects  of  the  model.  Upon  examination  of  raw  data 
for  any  of  these  measures,  it  becomes  clear  that  a  composite  implementation  is  more 
useful  than  a  measure  such  as  the  average  information  quality.  Furthermore,  a  statis¬ 
tic  such  as  the  raw  number  of  requests  that  entered  and  exited  the  system  provides 
limited  insight.  As  such,  the  quantity  measure  was  rolled  into  the  implementation 
of  the  other  measures  to  create  proportions  of  requests  that  met  quality,  timeliness, 
or  both  requirements.  The  use  of  proportions  in  this  way  also  makes  it  easier  to 
compare  various  alternatives  to  a  baseline  system.  Separating  these  proportions  into 
various  categories,  such  as  by  user  or  information  source,  also  provides  additional 
insight  into  the  system.  The  variables  that  collect  various  counts  and  sums  according 
to  the  expressions  in  Table  3.4  are  used  to  generate  these  proportions  at  the  end  of 
a  simulation  run. 

One  quality  measure,  collected  by 

Pct.QualMetJJsr_3  =  S.Num.QuaLUser{3)/S.Num.OutJJser{3), 


calculates  the  proportion  of  requests  by  user  3  that  met  the  quality  requirements. 
Note  that  this  is  independent  of  meeting  the  timeliness  requirement.  Additional 


quality  measures  are  implemented  similarly.  An  example  of  a  timeliness  measure, 


PctJ^imely^SrcJdl  =  S^NumJ^imely-Src{l)/S^NumJJut^Src{l) 

calculates  the  proportion  of  requests  needing  information  from  the  first  information 
source  that  meet  the  timeliness  requirement.  As  with  the  quality  measures,  addi¬ 
tional  timeliness  measures  are  implemented  similarly.  With  respect  to  meeting  both 
timeliness  and  qnality  reqnirements,  an  example  measure  is 

Pct-QT  .PrA  =  S-Num-QT.Priority{4)/S-Num-Out-Priority{4) 

which  calculates  the  proportion  of  items  with  Priority  JJ ser  =  4  that  meet  both  the 
timeliness  and  qnality  reqnirements.  Other  proportions  for  meeting  both  timeliness 
and  quality  requirements  are  implemented  similarly. 

In  addition  to  these  dehned  statistics,  some  measnres  such  as  resource  utiliza¬ 
tion,  average  wait  times  in  qnenes,  and  average  nnmber  of  items  in  the  system  are 
implemented  by  defanlt  within  Arena  so  long  as  the  option  to  collect  them  is  en¬ 
abled.  Because  of  preliminary  resnlts  for  wait  times  in  qnenes,  additional  statistics 
were  gathered  to  examine  the  total  wait  time  in  system  (i.e.,  combined  wait  times 
for  all  qnenes)  broken  down  by  priority.  For  example,  the  statistic 

S-AvgAV  aitT  ime^PrA  =  SPT  otAV  aitT  ime  ^Priority  {2)  /  S^NumJDut  ^Priority  {2) 

collects  the  average  wait  time  of  items  with  Priority  User  =  2  over  a  simnlation 
rnn.  Similar  statistics  are  implemented  for  the  other  priorities  and  the  maximnm 
wait  time  by  priority.  The  beneht  of  these  statistic  is  that  it  helps  to  provide  insight 
into  why  wait  times  for  quenes  might  appear  longer  than  expected.  Although  the 
total  wait  times  are  not  broken  ont  by  other  categories  like  the  proportion  statistics. 
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such  statistics  could  be  easily  implemented  by  the  simulation  user.  A  complete  list 
of  implemented  statistics  is  given  in  Appendix  E. 

3.5  Validation  and  Verification 

Early  in  the  model  development  process,  a  meeting  was  held  with  subject 
matter  experts  (SMEs)  to  ensure  that  the  initial  conceptual  model  was  correct. 
Based  on  the  results  of  this  meeting  and  further  research,  a  more  robust  conceptual 
model  was  developed.  As  the  conceptual  model  was  being  developed,  it  was  also 
drafted  in  Arena.  By  the  second  major  revision  of  the  conceptual  and  draft  Arena 
model,  a  description  was  written  and  sent  to  SMEs  for  review.  In  addition  to  the 
written  description,  basic  flowcharts  of  potential  entity  flow  throughout  the  model 
as  well  as  the  core  model  logic  for  determining  that  flow  were  submitted  to  SMEs 
for  review  (see  Appendix  C).  SME  comments  on  the  description  and  flowcharts  were 
used  to  update  the  conceptual  and  implemented  models.  Once  these  updates  were 
complete,  a  meeting  with  SMEs  was  held  to  conduct  a  low  level  walk-through  of  the 
model  implementation  in  Arena.  This  low  level  review  resulted  in  minor  revisions  to 
the  implemented  model.  Overall,  involving  SMEs  early  and  throughout  the  process 
helped  to  ensure  that  the  right  model  was  being  developed.  Multiple  reviews  of  the 
conceptual  and  implemented  models  also  helped  to  ensure  that  the  implementation 
was  correct. 

The  development  of  a  data  request  sheet  for  collecting  notional  or  real  data 
presented  an  additional  opportunity  to  verify  the  coding  of  the  model.  Furthermore, 
once  notional  data  was  constructed  for  the  sample  study,  inputing  data  into  the 
model  ensured  that  these  areas  of  the  model  were  correct.  With  notional  data  in  the 
model,  examination  of  simple  animation  and  output  statistics  was  used  to  determine 
if  the  model  was  working  as  expected.  The  use  of  animation  and  extensive  statistics 
allows  any  problems  to  be  easily  traced  to  various  portions  of  the  model.  As  an 
example,  some  portions  of  the  model  were  not  used  for  the  sample  study,  yet  an 
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incorrectly  set  input  variable  was  sending  items  to  that  portion  of  the  model  causing 
an  unexpected  bottleneck  in  the  system.  Some  of  the  sample  study  cases  discussed 
in  Section  3.7  were  used  to  stress  the  system.  Reasonable  results  provide  assurance 
that  the  model  is  correct.  Simulation  results  are  discussed  in  Section  4.3. 

3.6  Data  Requirements 

The  data  requirements  for  simulation  experiments  can  be  overwhelming.  Even 
in  a  high  level  model  such  as  this,  the  amount  of  information  required  is  extensive. 
The  data  required  for  this  model  can  be  divided  into  the  following  basic  categories: 
processing  (i.e.,  delay)  times,  resources,  effects  on  quality,  and  request  properties. 

The  simulation  requires  processing  and  delay  times  for  each  portion  of  the 
model  except  the  evaluation  submodel.  Everywhere  in  the  model  that  a  delay  is 
incurred,  a  resource  is  required.  Furthermore,  each  time  processing  of  some  sort 
occurs  on  an  RFI  and  update  is  made  to  the  actualized  quality  attribute,  QualA. 
The  request  properties  are  used  to  determine  when  and  what  type  of  RFIs  are 
generated.  A  complete  listing  of  the  required  data  for  a  simulation  study  is  given 
in  the  form  of  a  data  request  sheet  in  Appendix  D.  The  sheet  is  hlled  in  with  the 
data  used  for  the  sample  study  discussed  in  the  next  section.  One  key  aspect  to  note 
about  the  data  is  that  it  is  often  in  the  form  of  probability  distributions  rather  than 
just  raw  numbers. 

3.7  Sample  Study  Overview 

In  order  to  demonstrate  how  the  IPM  simulation  could  be  used  for  studies,  a 
sample  study  was  conducted.  Real  world  data  that  would  be  needed  to  populate 
the  model  was  not  available  due  to  its  sensitivity  and  security  classihcation.  As 
such,  notional  data  was  generated  for  the  sample  study  excluding  parts  of  the  model 
that  were  not  under  examination.  The  notional  data  was  used  as  a  baseline  for 
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comparative  study.  Prior  to  conducting  any  comparisons,  replication  truncation  and 
termination  points  were  selected.  The  analysis  to  determine  the  replication  length 
is  given  in  Section  4.2. 

In  addition  to  performing  analysis  on  the  baseline  simulation  results,  seven 
additional  simulations  were  performed  to  examine  various  aspects  of  the  model.  The 
cases  were  dehned  by  making  the  following  changes  to  the  baseline  model: 

1.  Change  Timely  ^Threshold  from  0  to  48  hours 

2.  Change  Timely  ^Threshold  from  0  to  12  hours 

3.  Change  QualMet ^Threshold  from  0  to  3 

4.  Change  QualMet ^Threshold  from  0  to  1 

5.  Increase  number  of  Additional  Requirements  by  50% 

6.  Increase  Exploitation  times  by  50% 

7.  Increase  Analysis  times  by  50% 

Note  that  only  one  change  was  made  for  each  case  and  any  prior  changes  were  reset 
to  the  baseline  settings.  Details  of  the  eight  simulations  are  given  in  Section  4.3. 
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4.  Results  and  Analysis 


4 . 1  Introduction 

Since  the  method  of  multiple  independent  replications  will  be  used  for  con¬ 
ducting  comparative  studies,  determination  of  appropriate  replication  length  is  im¬ 
portant.  A  single  long  run  of  the  baseline  system  described  by  the  notional  data 
given  in  Appendix  D  was  used  for  determining  an  appropriate  truncation  point  and 
replication  length.  The  process  of  determining  the  replication  truncation  and  termi¬ 
nation  points  is  discussed  in  Section  4.2.  Once  the  replication  length  was  determined 
the  simulation  was  run  for  25  replications  for  the  baseline  system  as  well  as  for  each 
of  seven  additional  cases  used  to  stress  various  parts  of  the  model.  The  results  and 
analysis  of  these  simulation  runs  are  discussed  in  Section  4.3. 

4-2  Replication  Truncation  and  Termination 

Although  multiple  replications  are  used  for  comparison  of  system  designs,  trun¬ 
cating  initial  data  for  statistics  collections  allows  removal  of  some  estimator  bias. 
Two  measures  were  examined  when  determining  truncation  points,  entity  total  time 
in  system  (TIS)  and  work  in  process  (WIP).  Ideally,  if  there  were  a  specihc  measure 
that  an  analyst  were  interested  in,  they  would  use  that  measure  for  determining 
replication  truncation  and  termination  points.  However,  since  the  sample  study  ex¬ 
amines  a  variety  of  measures,  the  TIS  and  WIP  measures  were  selected  since  they 
are  representative  of  the  system  in  general.  The  time  in  system  for  each  entity  that 
exited  the  system  and  the  number  of  items  being  serviced  and  in  queues  (i,.e.,  WIP), 
were  collected  in  a  text  hie  over  a  period  of  about  15  simulated  years  for  only  one 
replication.  Matlab  and  Octave  were  used  to  import  and  analyze  the  data.  The 
TIS  data  was  examined  directly  while  the  WIP  data  was  hrst  discretized  by  taking 
the  time-average  WIP  per  day.  It  was  determined  that  120  days  of  simulation  time 
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Moving  Average  of  Time  in  System  by  Observation 


Figure  4.1  Moving  average  plot  of  RFI  time  in  system. 


would  be  truncated  and  the  simulation  would  be  run  for  an  additional  4  years  (1460 
days)  of  simulated  time  for  a  total  replication  length  of  1580  days. 

4-2.1  Time  in  System 

In  the  process  of  determining  a  truncation  point,  a  moving  average  plot  of  TIS 
was  constructed  using  a  modified  version  of  the  algorithm  given  in  “The  Statistical 
Analysis  of  Simulation  Results”  [Welch,  1983:294].  The  moving  average  plot  for  TIS 
shown  in  Figure  4.1  was  constructed  using  a  batch  size  of  5000  observations.  Since 
multiple  replications  will  be  used  for  conducting  the  comparative  studies,  the  amount 
of  truncated  data  would  ideally  be  relatively  small.  Furthermore,  a  clear  truncation 
point  is  difficult  to  select.  As  a  compromise  a  truncation  time  of  30  days  was  selected, 
which  corresponds  to  about  5000  observations  which  is  indicated  in  the  hgure.  This 
allows  some  of  the  initialization  bias  to  be  accounted  for  without  throwing  away  too 
much  data. 
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Cumulative  Average  of  Time  in  System  by  Observation 


Figure  4.2  Cumulative  average  plot  of  RFI  time  in  system  using  truncated  data. 

To  determine  replication  length,  several  methods  were  employed.  One  method 
was  a  cumulative  average  plot  of  the  TIS.  This  plot,  given  in  Figure  4.2,  was  gener¬ 
ated  using  the  the  previous  data  with  the  hrst  5000  observations  truncated.  The  idea 
is  to  select  a  termination  point  for  replications  that  appears  to  be  near  steady  state. 
The  cumulative  average  appears  to  stabilize  after  about  60,000-75,000  observations. 
A  termination  point  of  two  years  of  simulated  time  beyond  truncation  was  selected 
which  corresponds  to  about  110,000  observations  which  is  marked  in  the  hgure. 

4-2.2  Work  in  Process 

After  the  WIP  data  was  discretized  by  taking  the  time-average  WIP  per  day, 
a  moving  average  plot  was  constructed  in  the  same  manner  as  before.  From  Figure 
4.3  a  truncation  point  can  be  selected  much  easier  than  for  the  TIS  data.  However, 
previously  selected  truncation  of  30  days  of  data  appears  to  be  insufficient.  As  such  a 
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Moving  Average  of  Time-Average  WiP  per  Day  by  Day 


Figure  4.3  Moving  Average  Plot  of  RFI  time-average  WIP  per  day. 


truncation  point  of  120  days  was  selected  with  keeping  the  amount  of  data  discarded 
relatively  small. 

After  truncating  120  days  of  the  data,  a  cumulative  average  plot  of  the  time- 
average  WIP  per  day  was  constructed.  This  chart  allows  one  to  visually  see  when 
the  variation  in  the  data  begins  to  have  a  reduced  affect  on  the  overall  mean  that  will 
be  calculated  for  each  replication.  The  2  year  replication  length  that  was  selected 
for  the  TIS  data  is  marked  in  Figure  4.4.  From  the  plot,  the  2  year  point  may  be  too 
short,  so  a  termination  point  of  4  years  was  selected  since  it  appears  to  be  near  the 
beginning  of  the  steady  state  for  the  cumulative  average.  Since  multiple  replications 
are  going  to  be  used,  it  should  not  be  necessary  to  select  a  termination  point  well 
into  steady  state.  If  multiple  replications  were  not  going  to  be  used,  then  a  longer 
replication  length  would  be  required  for  methods  such  as  batch  means. 
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Cumulative  Average  of  Time-Average  WIP  per  Day  by  Day 


Figure  4.4  Cumulative  Average  Plot  of  RFI  time-average  WIP  per  day  using  trun¬ 
cated  data. 

4-3  Simulation  Results 


The  simulation  was  run  for  each  of  the  following  cases  for  25  replications  with 
a  replication  length  of  1580  days  and  truncation  of  the  first  120  days: 


1.  Change  Timely  ^Threshold  from  0  to  48  hours 

2.  Change  Timely  .Threshold  from  0  to  12  hours 

3.  Change  QualMet. Threshold  from  0  to  3 

4.  Change  QualMet. Threshold  from  0  to  1 

5.  Increase  number  of  Additional  Requirements  by  50% 

6.  Increase  Exploitation  times  by  50% 

7.  Increase  Analysis  times  by  50% 

The  results  of  each  replication  were  stored  in  a  Microsoft  Access  database.  Arena 
version  5.00.02,  which  was  used,  stores  the  data  in  the  Access  97  database  format. 
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Several  queries  were  used  to  import  the  data  into  Microsoft  Excel  and  OpenOffice.org 
Calc  for  basic  analysis  and  chart  generation. 

The  large  quantity  and  categorical  nature  of  many  of  the  output  statistics 
lend  themselves  to  representation  with  bar  charts.  The  beneht  of  the  bar  charts  is 
that  they  allow  the  outputs  to  be  quickly  compared  for  practical  signihcance.  Some 
of  these  are  reported  and  discussed  in  the  following  sections.  Additional  detailed 
results  including  tables  of  point  estimates  with  associated  95%  conhdence  interval 
half-widths  are  reported  in  Appendix  F.  One  thing  to  note  in  general  is  that  the  half¬ 
widths  are  relatively  small  compared  to  the  point  estimates.  This  indicates  that  a 
sufficient  number  of  replications  was  selected  to  get  relatively  precise  point  estimates. 
However,  this  does  not  imply  that  the  point  estimates  are  the  true  expected  values 
since  they  do  not  give  an  indication  of  point  estimator  bias.  In  many  cases,  this  will 
allow  small  differences  in  point  estimates  to  be  statistically  signihcant,  but  does  not 
indicate  any  practical  signihcance.  As  such,  practical  signihcance  and  application 
will  be  the  focus  of  discussion. 

4.. 3.1  Baseline  System 

Since  the  baseline  model  contains  only  notional  data,  the  actual  numbers  for 
individual  statistics  have  little  meaning  in  themselves.  However,  they  allow  a  starting 
point  for  comparison  both  within  the  various  parts  of  the  model  as  well  as  with  other 
models.  Results  for  standard  process  performance  measures  will  be  discussed  hrst 
followed  by  the  additional  implemented  measures. 

One  statistic  examined  was  the  number  of  items  in  the  system  (NIS)  or  work 
in  process  (WIP).  This  includes  both  items  waiting  in  queues  and  being  serviced  by 
resources.  Figure  4.5  show  the  average  over  all  replications  and  the  average  of  the 
maximum  values  observed  in  each  replication.  One  thing  to  note  about  the  values  is 
that  the  average  maximum  value  is  more  that  twice  that  of  the  average.  However,  it 
does  not  indicate  how  often  the  NIS  is  near  the  average  maximum.  Since  there  are 
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Average  Number  of  Items  in  System 
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Figure  4.5  Baseline:  Average  and  average  maximum  number  of  RFIs  in  system 

relatively  few  resources  in  the  baseline  system,  the  NIS  can  be  attributed  primarily 
to  items  that  are  waiting  to  be  serviced.  This  measure  alone  gives  little  insight  into 
the  system  other  than  being  able  to  compare  overall  “efficiency”  since  it  depends  on 
both  the  rate  items  enter  and  leave  the  system.  However,  a  large  NIS  value  does  not 
necessarily  indicate  an  “inefficient”  system,  especially  considering  the  scale  of  a  high 
level  model  of  the  intelligence  process. 

In  addition  to  examining  WIP,  the  average  number  of  items  waiting  in  queues 
(NQ)  provides  more  insight  into  where  potential  bottlenecks  may  be  occurring.  For 
example,  in  Figure  4.6  the  average  number  of  items  waiting  for  collection  are  more 
than  half  the  average  WIP  value.  Alternatively,  the  average  NQ  for  some  of  the 
queues  is  essentially  zero  which  may  indicate  an  over  allocation  of  resources.  These 
NQ  values  are  dependent  upon  both  the  arrival  of  items  and  the  ability  of  resources 
to  process  those  items.  Moreover,  they  are  aggregate  values  for  multiple  types  and 
quantities  of  resources.  As  an  aggregate  measure,  it  can  be  easily  biased  high  or 
low  depending  on  the  layout  of  the  underlying  arrival  and  departure  rates  of  items 
in  the  queue.  The  average  of  the  maximum  values  of  NQ  for  each  replication  are 
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Baseline:  Average  number  of  RFIs  waiting  in  each  queue 


Average  Maximum  Number  Waiting  in  Queues 
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Figure  4.7  Baseline:  Average  maximum  number  of  RFIs  waiting  in  each  queue 
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Average  Wait  Time  in  Queues 
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Figure  4.8  Baseline:  Average  wait  time  (hours)  in  each  queue 

also  plotted  in  Figure  4.7.  In  general  for  the  baseline  system,  the  maximum  values 
are  about  twice  that  of  the  average  values.  As  with  the  average  maximum  WIP  this 
gives  no  indication  of  how  often  the  NQ  values  are  near  the  maximum.  It  is  also 
important  to  note  that  even  if  the  system  has  large  NQ  values  that  it  may  still  be 
performing  well  if  items  are  spending  a  relatively  small  amount  of  time  in  the  queue. 

Accordingly,  the  average  wait  time  (WT)  in  each  queue  is  plotted  in  Figure 
4.8  and  the  average  over  the  replications  of  the  maximum  wait  time  in  each  queue 
in  Figure  4.9.  It  is  interesting  to  note  that  the  relative  differences  in  average  WT 
for  the  various  queues  are  similar  to  that  of  the  NQ  measures.  However,  the  average 
maximum  WT  presents  a  different  picture.  Even  though  the  average  wait  time  in 
the  exploitation  queue  is  only  about  15  hours,  some  items  spent  around  1,700  hours 
waiting.  Based  on  the  input  data  for  the  baseline  simulation,  the  longest  an  item 
could  be  in  the  system  and  still  be  timely  is  120  hours,  so  those  items  were  certainly 
not  timely.  This  emphasizes  the  point  that  WT  is  as  critical  as  service  times  in  a 
system  where  timeliness  is  of  great  importance.  The  average  maximum  WT  does  not 
indicate  how  often  items  are  waiting  so  long  in  the  queue.  As  such,  the  impact  of  a 
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Figure  4.9  Baseline:  Average  maximum  wait  time  (hours)  in  each  queue 

few  items  that  wait  extensive  times  may  be  minimal.  Additional  timeliness  measures 
were  implemented  to  help  assess  such  an  impact.  Alternatively,  a  different  view  of 
WT  might  provide  better  insight  into  the  long  times. 

Partitioning  the  total  WT  in  system  (this  includes  all  queues)  by  priority  pro¬ 
vides  a  clearer  picture  of  what  is  happening.  The  Priority  ^Global  attribute  would 
be  ideal  for  this  decomposition  of  WT,  but  its  possible  values  are  not  constrained 
as  those  of  the  Priority  .User  attribute  are.  As  such,  the  total  WT  was  partitioned 
by  Priority^User.  For  this  study  there  is  no  difference  since  Priority  ^Global  is  set  to 
equal  Priority _User.  The  average  total  WT  by  priority  is  shown  in  Figure  4.10  and 
average  over  the  replication  maximum  WT  by  priority  in  Figure  4.11.  Given  that 
the  queues  are  generally  priority  based,  the  total  WT  by  priority  figures  appear  as 
one  would  expect.  The  highest  priority  items  spend  very  little  time  waiting  while 
the  low  priority  items  spend  a  long  time  waiting.  This  result  in  itself  aids  model 
verification. 

Another  standard  process  model  measure  is  resource  utilization.  The  average 
resource  utilization  for  the  various  resources  are  shown  in  Figures  4.12  and  4.13. 
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Figure  4.10  Baseline:  Average  total  wait  time  (hours)  by  Priority^User 
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Figure  4.11  Baseline:  Average  maximum  total  wait  time  (hours)  by  Priority_User 
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Figure  4.12  Baseline:  Average  resource  utilization 
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Figure  4.13  Baseline:  Average  utilization  of  user  analysts 


4-12 


Resource  utilization  is  dependent  upon  arrival  rates,  service  times,  and  number  of 
resources  available.  As  such,  the  large  NQ  values  above  are  likely  to  have  an  as¬ 
sociated  high  resource  utilization.  However,  small  NQ  values  can  also  have  high 
utilization  rates.  Unlike  the  queue  statistics  above,  the  resource  statistics  are  bro¬ 
ken  into  categories.  Values  over  one  occur  because  the  statistic  is  more  specifically 
scheduled  utilization,  some  resources  are  based  on  schedules,  and  the  schedule  policy 
is  set  to  have  resources  hnish  working  on  the  current  item  at  the  time  they  are  con¬ 
sidered  not  available.  For  example,  suppose  a  worker  is  scheduled  to  get  off  work  at 
1700  hours  and  is  in  the  middle  of  working  on  a  report.  With  the  currently  selected 
policy,  the  worker  would  work  overtime  to  complete  the  report  before  going  home 
resulting  in  a  scheduled  utilization  greater  than  one.  From  the  figures,  it  can  be 
seen  that  the  collection  resources  are  on  average  completely  utilized.  This  was  in¬ 
tentional  in  the  design  of  the  notional  data  and  is  also  reflected  in  the  large  NQ  and 
WT  values  associated  with  collection.  The  utilization  of  communications  resources 
after  collection  and  processing  resources  are  markedly  low.  It  is  unlikely  that  uti¬ 
lization  would  be  so  low  in  the  real  system  and  is  an  artifact  of  generating  notional 
data.  The  breakdown  of  Analyst.Src  indicates  that  for  some  sources  the  analysts 
are  over-tasked,  while  others  are  under-tasked.  The  AllSource Analysts  appear  to  be 
tasked  at  an  appropriate,  or  slightly  high,  level  even  though  they  are  used  for  both 
analysis  and  production.  User  analysts,  on  the  other  hand,  are  under  utilized,  but 
it  must  be  taken  into  consideration  that  user  integration  was  not  included  for  this 
study.  Utilization  coupled  with  the  previous  measures  provide  a  decent  picture  of 
the  system  performance.  The  insights  gathered  may  be  useful  in  hxing  problems 
with  the  system  but  alone  do  not  provide  a  solid  measure  of  system  performance 
impact  on  information  needs. 

To  address  the  inability  of  standard  process  performance  measures  to  ade¬ 
quately  provide  insight  into  the  effect  of  system  performance  on  information  needs, 
additional  measures  were  implemented.  The  additional  measures  are  presented  as 
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Figure  4.14  Baseline:  Average  proportion  of  requirements  met  by  Priority^User 


proportions  of  requirements  met.  These  measures  coupled  with  standard  process 
performance  measures  should  provide  adequate  insight  into  the  effect  of  system  per¬ 
formance  on  information  needs.  These  measures  are  decomposed  into  various  cate¬ 
gories  such  as  priority,  source,  and  user.  The  average  proportion  of  requirements  met 
by  Priority_User  is  shown  in  Figure  4.14.  As  would  be  expected  with  a  priority  based 
system,  the  highest  priority  requirements  are  on  average  almost  always  on  time.  As 
the  priority  goes  down,  so  does  the  proportion  that  meet  the  timeliness  requirement. 
This  could  be  partially  inferred  from  the  average  WT  by  priority  in  Figure  4.10.  The 
proportion  that  met  the  quality  requirement  is  also  represented.  It  is  interesting  to 
note  that  for  the  baseline  system  that  the  proportion  that  met  quality  requirements 
also  decreases  as  priority  decreases.  This  may  be  caused  partially  by  RFIs  that  never 
get  information  collected  and  partially  for  those  that  do  get  collected  by  not  reaping 
the  benehts  of  potential  quality  increases  in  the  remainder  of  the  system.  The  third 
proportion  is  that  of  meeting  both  the  timeliness  and  quality  requirements.  This 
measure  will  never  be  higher  than  the  lowest  of  the  timely  and  quality  proportions, 
but  it  will  not  necessarily  be  equal  to  the  lowest.  The  fact  that  the  proportion  of 
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Figure  4.15  Baseline:  Average  proportion  of  requirements  met  by  InfoSource 

meeting  both  requirements  drops  as  priority  decreases  aids  in  model  verification.  It 
is  interesting  to  note  that  most  of  priority  1  and  2  requests  are  met  with  a  relatively 
steep,  nearly  linear  drop  as  priority  continues  to  decrease. 

The  proportion  of  requirements  met  is  also  decomposed  by  InfoSource  and 
shown  in  Figure  4.15.  Since  much  of  the  required  information  for  running  the  sim¬ 
ulation  is  dependent  upon  the  InfoSource  attribute,  this  plot  could  be  helpful  in 
verihcation  of  input  information.  Note  however  that  a  source  that  provides  “per¬ 
fect”  information  quality  may  still  not  achieve  100%  for  meeting  quality  due  to  low 
priority  items  that  never  get  collected.  The  results  for  the  baseline  system  indicate 
that  the  hrst  source  meets  a  noticeably  lower  proportion  of  quality  requirements 
yet  a  comparable  proportion  of  timeliness  requirements.  The  net  result  is  a  lower 
proportion  of  both  requirements  being  met  than  for  the  other  sources.  This  result 
is  interesting  since  the  major  differences  between  the  hrst  and  second  sources  are 
related  to  various  times  to  complete  tasks,  rather  than  in  quality  updates  within 
the  system.  For  example,  the  collection  time  distributions  are  quite  different  even 
though  the  have  the  same  means.  The  hrst  source  has  exponential  times  and  the 
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Figure  4.16  Baseline:  Average  proportion  of  requirements  met,  standard  vs.  ad¬ 
ditional 

second  source  has  constant  times.  The  added  variation  in  collection  times  is  likely 
having  an  adverse  affect  on  meeting  quality  requirements  even  though  on  average  the 
same  proportion  of  timeliness  requirements  are  being  met.  One  could  do  an  entire 
study  investigating  the  effects  of  varying  collection  time  distributions. 

Another  way  the  requirements  are  partitioned  is  by  the  type  of  requirement, 
that  is,  standard  or  additional.  The  proportions  for  meeting  timeliness,  quality 
and  both  are  shown  in  Figure  4.16.  From  the  chart  it  can  be  seen  that  a  greater 
proportion  of  additional  requirements  are  met  than  standing  requirements.  One 
possible  reason  for  this  difference  is  that  standard  requirements  must  go  through 
the  collection  process  whereas  some  portion  of  the  additional  requirements  are  met 
without  requiring  collection. 

Lastly,  the  proportion  of  requirements  met  is  broken  down  by  user.  The  results 
for  the  baseline  system  are  given  in  Figure  4.17.  The  system  was  set  up  with  five 
users  who  had  identical  processing  requirements  for  items  found  in  a  library  search 
not  requiring  additional  collection  but  differed  in  the  amount  of  processing  required 
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Figure  4.17  Baseline:  Average  proportion  of  requirements  met  by  User 

for  any  additional  requirement  requiring  collection  and  standing  (i.e.  standard)  re¬ 
quirements.  Specifically,  for  any  requirement  needing  collection  User  1  required  all 
steps,  User  2  never  required  exploitation.  User  3  never  required  analysis.  User  4 
never  required  analysis  or  production,  and  User  5  never  required  exploitation  or 
production.  In  essence,  the  users  are  set  to  compare  various  architectures  in  a  sat¬ 
urated  system.  Rather  than  examine  a  TPED  like  system  independent  of  a  TPPU 
like  system,  it  would  be  more  realistic  that  some  requirements  may  always  need  to 
go  through  a  complete  process  while  other  requirements  may  be  able  to  skip  various 
portions  thereby  resulting  in  a  hybrid  system  that  cannot  be  adequately  character¬ 
ized  by  either  alone.  For  the  baseline  system,  it  can  be  seen  that  there  is  some 
variation  in  the  proportion  of  quality  requirements  met  is  but  it  is  fairly  consistent 
in  general.  As  one  might  expect,  fewer  of  the  timeliness  requirements  were  met  for 
User  1  than  the  other  users.  With  the  exception  of  User  1,  the  differences  between 
the  proportions  for  meeting  both  requirements  may  or  may  not  be  practically  signif¬ 
icant  since  the  differences  are  at  most  a  few  percent.  Conducting  a  paired-t  test  on 
the  difference  between  User  1  and  User  5  for  Timely,  QualMet,  and  Both  as  given 
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Figure  4.18  BL  vs  Cl:  Average  and  average  maximum  work  in  process 

in  Figure  4.17  results  in  p- values  that  are  essentially  zero  (6.19E-38,  8.93E-29,  and 
1.67E-34,  respectively).  A  selection  of  p- values  for  other  t-test  comparisons  of  User 
1  and  User  5  for  this  and  other  cases  is  given  in  Appendix  F.9. 


4-3.2  Case  1:  Extreme  Timely  Threshold 

Case  1  involved  changing  the  value  of  the  Timely  ^Threshold  variable  from  0  to 
48  hours.  This  has  the  same  effect  as  extending  all  TimeR  values  by  48  hours  since 
the  threshold  value  is  used  whenever  a  timeliness  check  is  done.  It  is  unlikely  that 
such  extensions  for  all  requests  would  occur  in  real  life,  but  this  case  is  still  useful  for 
testing  the  system.  Rather  than  restate  all  of  the  statistics  reported  for  the  baseline 
system  in  Section  4.3.1,  only  a  selected  portion  that  provide  interesting  results  will 
be  reported  here.  Additional  details  for  Case  1  results  can  be  found  in  Appendix 
F.2. 

The  hrst  statistic  of  interest  is  the  average  number  of  items  in  system  at  any 
time  or  WIP.  A  plot  of  the  average  and  average  maximum  values  for  the  Baseline 
(BL)  and  Case  1  (Cl)  results  are  shown  in  Figure  4.18.  As  can  be  seen  in  the 
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Figure  4.19  BL  vs  Cl:  Average  number  waiting  in  queues 

figure,  the  WIP  has  doubled  for  both  the  average  and  the  average  maximum.  This 
result  is  not  unexpected  as  items  are  allowed  to  remain  in  the  system  longer,  the 
number  waiting  would  be  expected  to  increase.  The  increase  in  the  number  of  RFIs 
waiting  in  the  various  queues  can  be  seen  in  Figure  4.19.  Interestingly  enough,  there 
is  no  practically  significant  difference  in  the  resource  utilization.  This  is  partially 
reflected  in  the  fact  that  wait  times  for  lower  priority  items  has  also  increased  (see 
Figure  4.20).  Even  with  the  48  hour  extension,  the  increased  wait  times  for  the  lower 
priority  items  significantly  exceed  the  required  time.  Since  the  RFIs  are  waiting  so 
long  before  being  serviced,  they  are  thrown  out  of  the  system  and  do  not  cause 
increased  resource  utilization. 

If  we  examine  the  proportion  of  requirements  met,  the  added  time  for  meeting 
requirements  provides  little  benefit.  The  proportion  of  meeting  both  quality  and 
timeliness  requirements  for  each  user  is  shown  in  Figure  4.21.  Only  a  marginal 
increase  is  seen  which  likely  has  no  practical  significance.  The  results  for  partitioning 
the  proportion  of  requirements  met  by  Priority^User  are  shown  in  Figure  4.22  and 
indicate  an  interesting  change  from  the  baseline  results.  There  is  no  practically 
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Figure  4.20 


BL  vs  Cl:  Average  total  wait  time  (hours)  by  Priority.User 


Average  Proportion  of  Requirements  Met  by  User 


User 


□  bl 

■  C1 


Figure  4.21 


BL  vs  Cl:  Average  proportion  of  requirements  met  by  User 
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Figure  4.22  BL  vs  Cl:  Average  proportion  of  requirements  met  by  Priority^User 

significant  change  in  the  proportion  of  priority  1  and  2  requirements  met,  but  there 
is  an  increase  in  the  proportion  of  priority  3  requirements  met.  This  increase  is  offset 
by  a  decrease  in  the  proportion  of  priority  4  and  5  requirements  met. 

Overall,  the  addition  of  48  hours  to  the  threshold  for  determining  timeliness 
had  a  negative  impact  on  the  system.  The  only  potential  beneht  was  an  increase  in 
the  proportion  of  priority  3  requirements  met;  however,  this  comes  at  a  great  cost. 
The  disadvantages  of  increased  WIP  and  wait  times  as  well  as  meeting  essentially 
none  of  the  priority  5  requirements  likely  outweigh  such  a  minor  beneht.  From  the 
perspective  of  the  users  there  is  no  real  beneht,  especially  when  one  considers  the 
fact  that  requirements  may  be  met  up  to  two  days  later  than  in  the  baseline  system. 


4-3.3  Case  2:  Increased  Timely  Threshold 

Case  2  involved  changing  the  value  of  the  Timely -Threshold  variable  from  0  to 
12  hours.  Similar  to  Case  1,  this  has  the  same  ehect  as  extending  all  TimeR  values 
by  12  hours  since  the  threshold  value  is  used  whenever  a  timeliness  check  is  done. 
Although  it  is  unlikely  that  such  extensions  would  occur  for  all  items  in  real  life,  it 
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Figure  4.23  BL  vs  C2:  Average  and  average  maximum  work  in  process 

is  more  reasonable  that  Case  1.  A  12  hour  extension  would  be  analogous  to  stating 
that  if  an  item  was  due  by  close  of  business  one  day  then  it  is  OK  to  send  it  not  later 
than  start  of  business  the  next  day.  Rather  than  restate  all  of  the  statistics  reported 
for  the  baseline  system  in  Section  4.3.1,  only  the  results  reported  for  Case  1  will  be 
re-examined  here.  Additional  details  for  Case  2  results  can  be  found  in  Appendix 
F.3. 

The  first  statistic  revisited  is  the  average  number  of  items  in  system  at  any 
time  or  WIP.  A  plot  of  the  average  and  average  maximum  values  for  the  BL,  Cl,  and 
Case  2  (C2)  results  are  shown  in  Figure  4.23.  As  might  be  expected,  the  WIP  for  C2 
is  between  that  of  the  BL  and  Cl.  The  C2  increase  over  the  BL  results  appears  to  be 
a  little  less  than  1/4  of  the  Cl  increase,  so  the  change  in  WIP  seems  to  be  somewhat 
proportional  for  the  region  of  the  threshold  examined.  In  addition  to  the  NIS  results, 
the  results  for  number  waiting  in  queues  might  also  be  expected  to  behave  similarly. 
However,  the  increase  in  the  number  of  RFIs  waiting  in  the  various  queues  for  C2 
(see  Figure  4.24)  appear  to  be  at  essentially  the  same  levels  as  Cl.  As  with  Cl,  there 
is  no  practically  signihcant  difference  in  the  resource  utilization  between  the  BL  and 
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Figure  4.24  BL  vs  C2:  Average  number  waiting  in  queues 


C2.  The  lack  of  a  change  in  the  number  waiting  in  queues  is  particularly  interesting 
given  that  the  wait  times  for  lower  priority  items  have  decreased  compared  to  Cl 
(see  Figure  4.25).  Unlike  the  WIP  values,  the  increase  in  wait  times  for  Cl  and  C2 
do  not  appear  to  be  proportional  to  the  change  in  timeliness  threshold.  Even  so, 
the  large  average  wait  times  for  priority  4  and  5  items  for  C2  give  some  indication 
that  there  is  likely  to  be  a  relatively  low  proportion  of  requirements  met  for  those 
priorities. 

Examining  the  proportion  of  requirements  met  by  Priority -User  in  Figure  4.26 
indicate  that  the  C2  results  have  a  similar  pattern  as  the  Cl  results.  As  expected, 
the  C2  results  fall  between  the  BL  and  Cl  results.  It  appears  that  some  proportion  of 
the  meeting  the  priority  3  requirements  has  been  redistributed  to  the  lower  priority 
items.  Additionally,  there  is  still  no  practically  signihcant  difference  between  the 
priority  1  and  2  proportions.  Re-examining  the  proportion  of  requirements  met  for 
each  User  (see  Figure  4.27)  indicates  that  there  is  still  no  practically  significant 
difference  in  the  proportions  of  requirements  met. 
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Figure  4.25  BL  vs  C2:  Average  total  wait  time  (hours)  by  Priority.User 


□  bl 

■  C1 
[□C2 


Priority 


Figure  4.26 


BL  vs  C2:  Average  proportion  of  requirements  met  by  Priority _User 
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Figure  4.27  BL  vs  C2:  Average  proportion  of  requirements  met  by  User 

Overall,  the  addition  of  12  hours  to  the  threshold  rather  than  48  hours  for 
determining  timeliness  had  a  reduced  impact  on  the  system.  There  is  only  a  minor 
potential  benefit  in  an  increase  in  the  proportion  of  priority  3  requirements  met; 
however,  this  still  comes  at  some  cost.  The  same  disadvantages  that  were  present 
for  Cl  are  still  present  for  C2,  except  at  reduced  levels.  From  the  perspective  of  the 
users  there  is  no  real  benefit,  especially  when  one  considers  the  fact  that  requirements 
may  be  met  later  than  in  the  baseline  system.  On  the  other  hand,  if  users  can  live 
with  later  results,  they  may  receive  additional  middle  priority  items  but  only  at  the 
cost  of  lower  priority  items.  That  may  be  an  acceptable  trade-off  for  the  user  but 
only  if  the  remainder  of  the  system  can  feasibly  handle  the  increased  number  of  items 
in  the  system. 


4-3.4  Cases  3  and  4-  Increased  Quality  Threshold 


Case  3  involved  changing  the  value  of  the  QualMet_Threshold  variable  from  0 
to  3  and  Case  4  involved  changing  the  QualMet.Threshold  from  0  to  1.  This  should 
have  a  similar  effect  as  either  decreasing  QualR  or  increasing  QualA  by  3  or  1  for 
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Figure  4.28  BL  vs  C3,  C4:  Average  and  average  maximum  work  in  process 


Case  3  and  4,  respectively.  However,  changing  the  threshold  value  does  not  actually 
alter  the  QualR  or  QualA  attributes  and  would  not  be  expected  to  impact  timeliness 
as  much  as  altering  those  values  since  Equation  3.3  is  only  used  when  assessing 
library  search  results  and  when  assessing  items  as  they  leave  the  system.  As  with 
Case  1,  this  change  for  Case  3  is  unrealistic  but  is  useful  for  stressing  the  system. 
The  change  for  Case  4  may  be  more  realistic  for  selected  items,  but  is  unlikely  to  be 
as  useful  for  studies  as  altering  the  QualR  or  QualA  attributes.  A  value  of  0.5  for 
QualMeRThreshold  would  be  similar  to  rounding  the  QualA  attribute  to  the  nearest 
integer  prior  to  comparing  it  with  QualR  which  may  be  the  largest  reasonable  value 
for  the  threshold  unless  one  is  modeling  user  overstatement  of  requirements.  Only 
a  selected  portion  of  the  statistics  reported  for  the  baseline  system  in  Section  4.3.1 
that  provide  interesting  results  will  be  reported.  Additional  details  for  Case  3  and 
Case  4  results  can  be  found  in  Appendices  F.4  and  F.5,  respectively. 

The  average  number  of  RFIs  in  the  system  for  the  Baseline  (BL),  Case  3  (C3), 
and  Case  4  (C4)  is  shown  in  Figure  4.28.  As  expected,  there  is  minimal  impact 
and  to  the  WIP  for  either  C3  or  C4  and  no  practical  difference.  The  negligible 
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Figure  4.29  BL  vs  C3,  C4:  Average  proportion  of  requirements  met  by  Prior¬ 
ity  _  User 

apparent  difference  is  likely  due  to  a  small  increase  in  the  proportion  of  items  that  met 
requirements  in  a  library  search;  however,  no  statistics  were  collected  to  specifically 
test  this. 

Examining  the  impact  of  increasing  QualMet^Threshold  on  the  proportion  of 
requirements  met  by  Priority_User  (see  Figure  4.29  )  indicates  that  there  is  only 
minimal  and  practically  insignificant  increase  for  both  C3  and  C4.  The  results  of 
partitioning  the  proportion  of  requirements  met  by  InfoSource  and  User  (Figures 
4.30  and  4.31,  respectively)  indicate  a  similar  phenomenon. Even  though  the  is  no 
practical  difference  between  C3  and  C4  for  User  1,  there  is  a  statistically  significant 
difference;  a  paired-t  comparison  results  in  a  p-value  of  about  0.0042.  Similarly,  the 
difference  between  C3  and  C4  for  User  5  is  statistically  but  not  practically  different; 
a  paired-t  test  results  in  a  p-value  of  about  0.0385.  Additional  results  for  selected 
paired-t  comparisons  are  given  in  Appendix  F.9. 

The  only  exception  is  an  increase  in  the  proportion  of  requirements  met  for 
InfoSource  1  with  no  practical  difference  between  C3  and  C4.  A  closer  examination 
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Figure  4.30  BL  vs  C3,  C4:  Average  proportion  of  requirements  met  by  InfoSource 
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Figure  4.31 


BL  vs  C3,  C4:  Average  proportion  of  requirements  met  by  User 
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Figure  4.32  Case  4:  Average  proportion  of  requirements  met  by  InfoSource 

of  some  of  C4  results  will  provide  a  better  understanding  of  why  a  greater  increase 
in  the  proportion  of  requirements  met  is  not  realized. 

The  breakdown  of  the  C4  requirements  by  timeliness  and  quality  shown  in 
Figure  4.32  reveals  the  lack  of  a  signihcant  increase  in  the  proportion  of  requirements 
met.  As  it  can  be  seen,  quality  has  been  discounted  enough  that  timeliness  has 
become  the  limiting  factor  for  meeting  both  requirements.  Re-examination  of  the 
figures  in  Section  4.3.1  will  reveal  that  the  proportion  for  meeting  the  timeliness 
requirement  is  in  most  cases  close  to  that  for  meeting  both  requirements  resulting 
in  no  practical  improvement. 

4-3.5  Case  5:  Increased  Number  of  Additional  Requirements 

Case  5  (C5)  involved  increasing  the  number  of  additional  requirements  by 
about  50%.  Rather  than  increasing  the  rate  at  at  which  batch  arrivals  occur,  the 
number  of  requests  per  arrival  was  changed  from  a  DISC  (.5, 10, 1, 15)  distribution 
to  a  DISC  (.5, 15, 1,23)  distribution  for  all  users,  where  the  DISC  (■)  function  de¬ 
scribes  a  discrete  cumulative  distribution  function.  This  has  the  effect  of  modeling 
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Figure  4.33  BL  vs  C5:  Average  and  average  maximum  work  in  process 

a  sustained  surge  in  additional  requirements  from  all  users.  The  surge  is  sustained 
because  the  distribution  of  requests  per  arrival  and  time  between  arrivals  were  set 
to  remain  the  same  throughout  the  entire  simulation.  As  with  the  previous  cases, 
only  a  portion  of  the  statistics  reported  for  the  baseline  system  in  Section  4.3.1  that 
provide  interesting  results  will  be  revisited  in  this  section.  Additional  details  for 
Case  5  results  can  be  found  in  Appendix  F.6. 

Since  the  number  of  entities  arriving  in  the  system  has  increased,  it  should  come 
as  no  surprise  that  the  average  number  of  entities  in  the  system  has  also  increased. 
The  average  and  average  maximum  WIP  shown  in  Figure  4.33  indicate  that  there 
was  more  than  a  fourfold  increase  on  average.  Such  a  large  increase  gives  some 
indication  that  there  is  a  limiting  factor  or  bottleneck  in  the  model.  Examining  the 
average  number  of  items  waiting  in  various  queues  (see  Figure  4.34)  indicates  that 
the  limiting  factor  is  collection.  The  large  increase  is  likely  due  to  the  fact  that  the 
collection  resources  were  already  tasked  at  capacity  prior  to  increasing  the  number 
of  requirements  (see  Figure  4.12).  Although  it  is  interesting  that  the  size  of  the 
exploitation  queue  appears  to  be  slightly  smaller,  the  difference  is  likely  not  of  any 
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Figure  4.34  BL  vs  C5:  Average  number  of  RFIs  waiting  in  queues 

practical  significance.  Such  a  large  average  queue  size  and  average  WIP  indicate  that 
RFIs  will  spend  longer  in  the  system.  Examining  the  average  wait  time  by  priority 
(see  Figure  4.35)  shows  that  the  low  priority  items  are  spending  tremendously  more 
time  waiting  on  average  for  C5  than  for  the  BL.  With  such  large  average  wait  times, 
one  would  hardly  expect  many  of  the  low  priority  requirements  to  be  met.  In  fact, 
examining  the  average  proportion  of  requirements  met  by  Priority -User  (see  Figure 
4.36)  confirms  the  expected  result.  Furthermore,  this  figure  highlights  an  interesting, 
but  not  unreasonable  result.  Since  the  system  has  become  overloaded  with  requests, 
it  can  only  maintain  the  proportion  of  requirements  met  for  the  top  priority  items. 
The  proportion  of  requirements  met  for  all  other  priorities  has  dropped  significantly. 
Since  the  users  have  flooded  the  system  with  additional  requests,  and  the  collection 
resources  cannot  keep  up,  a  lower  proportion  of  their  requirements  are  being  met  (see 
Figure  4.37).  The  reduction  in  the  proportion  of  requirements  met  by  InfoSource  is 
also  exhibited  in  Figure  4.38. 

Overall,  the  result  of  a  50%  increase  in  additional  user  requirements  overloaded 
the  system.  The  overload  system  was  only  able  to  keep  up  with  the  highest  priority 
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igure  4.35  BL  vs  C5:  Average  total  wait  time  by  Priority_User 
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BL  vs  C5:  Average  proportion  of  requirements  met  by  Priority.  User 
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4.37  BL  vs  C5:  Average  proportion  of  requirements  met  by  User 
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Figure  4.38  BL  vs  C5:  Average  proportion  of  requirements  met  by  InfoSource 
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items.  Furthermore,  the  general  effect  was  that  a  smaller  proportion  of  requests 
were  satisfied.  One  could  also  examine  the  raw  number  of  requirements  that  were 
satisfied  (this  was  not  done),  but  that  might  give  an  overly  optimistic  view  of  the 
overloaded  system.  This  or  a  similar  increase  in  requirements  generated  can  be  used 
to  find  bottlenecks  or  other  weak  points  in  the  system  if  none  are  initially  apparent 
from  a  given  baseline. 

4-3.6  Case  6:  Increased  Exploitation  Times 

Case  6  (C6)  involved  increasing  all  exploitation  times  by  50%.  Rather  than 
alter  the  distribntions  as  was  done  for  Case  5,  a  mnltiplicative  factor  of  1.5  was 
added  to  the  block  that  assigns  the  delay  for  exploitation.  This  has  the  effect  of 
increasing  the  delay  for  any  RFI  that  nndergoes  exploitation.  Partially  done  to 
stress  the  system,  this  sort  of  adjustment  conld  be  examined  if  there  were  reason 
to  believe  that  the  exploitation  times  were  underestimated.  However,  snch  a  large 
increase  can  artificially  induce  a  bottleneck  in  the  system  as  was  the  intent  with 
this  case.  Similar  to  previous  cases,  only  interesting  resnlts  will  be  re-examined  here 
rather  than  the  complete  set  of  statistics  reported  for  the  baseline  system  in  Section 
4.3.1.  Additional  details  for  Case  6  results  can  be  fonnd  in  Appendix  F.7. 

As  expected,  the  increased  time  for  exploitation  has  increased  the  total  nnmber 
of  items  in  the  system  (see  Fignre  4.39)  The  majority  of  this  increase  is  due  to  the 
number  of  items  waiting  in  the  exploitation  qnene  (see  Fignre  4.40).  As  with  C5,  the 
large  total  nnmber  of  items  in  system  brings  into  qnestion  whether  or  not  the  system 
is  any  longer  in  steady  state  at  the  end  of  the  4  year  replications  or  if  a  steady  state 
any  longer  exists.  A  more  detailed  analysis  of  WIP  would  need  to  be  accomplished 
in  order  to  make  this  determination.  Along  with  the  larger  nnmber  of  items  in  qnene 
and  in  system,  we  would  expect  higher  wait  times.  Examining  the  wait  times  by 
priority  shows  that  the  lower  priority  items  are  indeed  waiting  longer  on  average. 
However,  it  is  interesting  to  note  that  in  Fignre  4.41  that  priority  4  items  are  waiting 
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Figure  4.39  BL  vs  C6:  Average  and  average  maximum  work  in  process 
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Figure  4.40  BL  vs  C6:  Average  number  of  RFIs  waiting  in  queues 
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Figure  4.41  BL  vs  C6:  Average  total  wait  time  (hours)  by  Priority -User 


longer  than  priority  5  items  on  average.  The  average  maximum  for  priority  5  items 
is  nonetheless  higher  than  the  average  maximum  for  priority  4  items.  One  possible 
explanation  for  this  is  that  priority  5  items  that  are  waiting  in  the  exploitation  queue 
may  never  get  serviced  and  thereby  will  never  exit  the  system,  thereby  biasing  the 
results.  Furthermore,  the  average  proportion  of  requirements  met  by  priority  follows 
the  expected  trend  of  a  lower  proportion  of  priority  5  items  being  met  than  priority 

4  items  (see  Figure  4.42).  Similar  to  C5,  the  system  can  only  keep  up  with  the 
top  priority  items.  Priority  2  and  3  items  have  suffered  a  practically  significant 
decrease  in  proportion  of  requirements  met,  whereas  the  proportion  of  priority  4  and 

5  items  did  not  change  in  a  practically  significant  manner.  Breaking  the  proportion 
of  requirements  met  down  by  InfoSource  (see  Figure  4.43)  provides  some  interesting 
results.  From  the  hgure  it  can  be  seen  that  there  is  no  practically  signihcant  difference 
in  the  proportion  of  requirements  met  for  sources  1  and  2,  but  sources  3  and  4  have 
incurred  a  significant  decrease.  The  reason  for  what  might  seem  like  a  discrepancy 
can  be  found  by  examining  the  utilization  of  the  AnalystSrc  resources  involved  in 
performing  exploitation.  From  the  chart  in  Figure  4.44  it  can  be  seen  that  there 
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BL  vs  C6:  Average  proportion  of  requirements  met  by  Priority _User 
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Figure  4.43  BL  vs  C6:  Average  proportion  of  requirements  met  by  InfoSource 
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Figure  4.44  BL  vs  C6:  Average  utilization  of  Analyst^Src  resources 

was  additional  capacity  for  sources  1  and  2  whereas  the  resources  were  already  at 
maximum  utilization  for  sources  3  and  4.  From  the  user  perspective,  we  see  what 
may  be  an  initially  surprising  picture  in  Figure  4.45.  However,  as  noted  Section 
4.3.1  discussing  the  BL  system  that  users  2  and  5  do  not  require  exploitation  except 
for  possibly  a  small  portion  of  additional  requirements.  That  distinction  makes  it 
clear  as  to  why  users  1,3,  and  4  have  seen  a  practically  signihcant  reduction  in  their 
proportion  of  requirements  met  while  users  2  and  5  have  seen  no  signihcant  change. 

Overall,  the  50%  increase  in  delay  times  for  exploitation  successfully  intro¬ 
duced  a  bottleneck  into  the  system,  albeit  unrealistic.  The  end  result  of  a  higher 
WIP,  longer  wait  times,  and  lower  proportion  of  requirements  met  is  consistent  with 
expected  results.  The  only  possibly  surprising  result  was  the  average  wait  times  for 
priority  4  RFIs  exceeded  the  average  wait  times  for  priority  5  RFIs.  One  possible, 
but  not  substantiated,  explanation  was  that  priority  5  RFIs  are  never  leaving  the 
exploitation  queue,  thereby  biasing  the  result. 
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Figure  4.45  BL  vs  C6:  Average  proportion  of  requirements  met  by  User 

4.3.7  Case  7:  Increased  Analysis  Times 

Case  7  was  created  by  increasing  all  analysis  times  by  50%.  This  was  accom¬ 
plished  in  a  manner  similar  to  Case  6.  A  simple  multiplicative  factor  of  1.5  was  added 
to  the  block  that  assigns  the  delay  for  analysis.  This  indiscriminately  increases  the 
analysis  times  for  all  RFIs  that  require  analysis.  The  primary  focus  is  again  to  stress 
the  system  and  introduce  an  artificial  bottleneck.  Only  a  portion  of  the  statistics 
reported  for  the  baseline  system  in  Section  4.3.1  that  provide  interesting  results  will 
be  revisited  here.  Additional  details  for  Case  7  results  can  be  found  in  Appendix 
F.8. 

Since  the  intent  of  this  case  was  to  introduce  a  bottleneck  into  the  system,  the 
hrst  statistic  examined  was  the  average  WIP.  As  can  be  seen  in  Figure  4.46  there 
is  an  expected  increase  in  the  both  the  average  and  average  maximum  number  of 
items  in  system.  Even  though  there  appears  to  be  an  increase,  the  number  of  items 
in  the  C7  system  are  only  a  little  less  than  twice  that  of  the  BL  system.  Focusing 
in  on  the  average  number  of  items  in  the  various  queues  reveals  that  there  is  an 
expected  backlog  of  items  in  the  analysis  queue  (see  Figure  4.47).  Corresponding 


4-39 


Average  Number  of  Items  in  System 


1600.0  - 

1500.0  - 

1400.0  - 

1300.0  - 

1200.0  - 

£ 

0) 

1100.0  - 

1000.0  - 

c/} 

c 

900.0  - 

(/) 

800.0  - 

E 

o> 

700.0  - 

600.0  - 

500.0  - 

400.0  - 

300.0  - 

200.0  - 

100.0  - 

0.0  - 

1600 

1500 

1400 

1300 

1200 

1100 

1000 

900 

800 

700 

600 

500 

400 

300 

200 

100 

0 


Average 


Max 


□  bl 

■  C7 


Figure  4.46  BL  vs  C7:  Average  and  average  maximum  work  in  process 
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Figure  4.47 


BL  vs  C7:  Average  number  of  RFIs  waiting  in  queues 
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Figure  4.48  BL  vs  C7:  Average  utilization  of  AllSourceAnalyst^Spec  resources 

to  the  substantial  queue  size,  the  all  source  analyst  resources  are  100%  utilized  on 
average  for  C7  (see  Figure  4.48).  Note  that  this  does  not  appear  to  be  a  practically 
significant  change  from  the  BL,  but  a  small  change  near  peak  utilization  may  in 
reality  be  significant.  It  is  interesting  to  note  that  even  though  the  production  process 
uses  the  same  resources  as  the  analysis  process,  the  production  queue  does  not  have 
a  large  queue  size  on  average.  Since  the  production  process  has  equal  priority  for 
using  the  all  source  analyst  resources,  this  may  be  due  to  a  reduced  arrival  rate  of 
items  to  the  production  process.  Related  to  the  increased  queue  sizes  and  number 
in  system,  the  average  wait  time  by  priority  shows  that  the  average  wait  time  for 
lower  priority  items  has  increased  (see  Figure  4.49).  For  the  lower  priority  items,  it 
appears  that  the  C7  wait  times  have  nearly  doubled  compared  to  the  BL  wait  times. 
Consequently,  as  can  be  seen  in  Figure  4.50  the  proportion  of  requirements  met  by 
priority  has  signihcantly  decreased  for  all  priorities  except  the  top  priority  items. 
When  the  proportion  of  requirements  met  is  broken  down  by  InfoSource,  Figure  4.51 
indicates  that  a  practically  signihcant  decrease  in  the  proportion  of  requirements 
met  is  realized  for  all  sources.  This  is  not  unexpected  since  the  analysis  times  were 
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Figure  4.49 


BL  vs  C7:  Average  total  wait  time  (hours) 
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BL  vs  C7:  Average  proportion  of  requirements  met  by  Priority _User 
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Figure  4.51  BL  vs  C7:  Average  proportion  of  requirements  met  by  InfoSource 

increased  without  regard  to  the  source  of  information.  From  the  users  perspective, 
only  User  4  maintained  the  proportion  of  requirements  met  in  C7  that  was  obtained 
in  the  BL.  Recalling  the  user  setup  discussed  in  Section  4.3.1,  User  4  was  the  only 
user  that  required  neither  analysis  nor  production.  All  other  users  required  at  least 
one  of  analysis  or  production  and  as  a  result  of  the  roughly  100%  analyst  utilization 
shown  in  Figure  4.48,  ended  up  with  a  fewer  proportion  of  requirements  met. 

Overall,  the  results  of  Case  7  are  consistent  with  expectations  given  the  50% 
increase  in  analysis  times.  As  expected,  increased  WIP,  larger  analysis  queue  size, 
and  longer  wait  times  for  lower  priority  items  were  observed.  Given  the  common 
resource  usage  between  the  analysis  and  production  phases,  a  smaller  proportion  of 
requirements  were  met  even  for  those  users  that  did  not  require  analysis  so  long  as 
they  required  production. 


4 . 4  Summary 


The  selected  replication  length  and  termination  points  allowed  for  an  assess¬ 
ment  of  the  baseline  system  near  steady  state  responses.  The  seven  cases  provided 
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Figure  4.52  BL  vs  C7:  Average  proportion  of  requirements  met  by  User 

simplistic  cases  to  stress  the  system  to  aid  in  model  verification.  Although  more 
complex  and  less  extreme  changes  may  be  more  realistic,  they  may  also  provide  re¬ 
sults  that  are  not  as  easy  to  interpret.  The  results  of  the  seven  cases  consistently 
impacted  the  system  in  ways  that  would  be  expected  given  the  nature  of  each  indi¬ 
vidual  change.  In  a  few  cases  there  were  some  slightly  unexpected  results  but  they 
were  not  unacceptable.  In  general  an  explanation  of  the  unexpected  results  could 
be  found  by  considering  the  interaction  of  other  aspects  of  the  model.  The  favor¬ 
able  results  of  the  seven  cases  lay  the  foundation  for  investigation  of  more  complex 
interactions  within  the  model. 


4-44 


5.  Conclusions 

This  chapter  will  provide  a  general  discussion  of  the  Intelligence  Process  Model.  Both 
strengths  and  limitations  of  the  model  will  be  briefly  reexamined.  Additionally,  some 
potential  areas  of  application  for  the  model  and  simulation  will  be  presented.  Finally, 
some  areas  of  future  research  will  be  introduced. 

5.1  Discussion 

The  initial  focus  of  this  study  was  on  developing  a  high  level  simulation  model 
to  address  the  need  to  do  quick  turn  studies  on  the  ability  of  intelligence  processes 
to  support  ISR  systems.  A  review  of  prior  efforts  in  modeling  intelligence  process 
generally  focused  on  a  specihc  implementation  such  as  TPED  or  required  too  much 
detail  for  a  high  level  model.  As  a  result,  the  Intelligence  Process  Model  was  devel¬ 
oped  primarily  from  the  high  level  perspective  of  the  intelligence  cycle  presented  in 
Joint  Publications  as  well  as  SME  input.  The  beneht  of  this  approach  is  that  it  is 
grounded  on  documented  processes  while  taking  a  top  down  view.  As  a  high  level 
model,  the  IPM  requires  less  information  to  populate  than  a  highly  detailed  model. 
Even  so,  the  amount  of  information  required  to  populate  the  model  is  still  exten¬ 
sive.  One  challenge  the  high  level  perspective  provides  to  analysts  is  appropriately 
aggregating  existing  detailed  information. 

The  IPM  was  also  developed  in  a  modular  fashion  with  emphasis  on  a  core 
module  tying  all  the  other  pieces  together.  This  allows  for  additional  detail  to 
be  later  added  to  any  particular  submodel  with  minimal  or  no  changes  required 
for  the  rest  of  the  model.  Additionally,  many  details  were  organized  in  arrays  of 
information  to  simplify  the  visual  layout  and  aid  in  customization  of  the  model  for 
particular  systems  and  scenarios.  One  beneht  of  using  arrays  is  that  the  probability 
distributions  and  information  for  a  selected  scenario  are  input  in  only  a  few  locations 
rather  than  requiring  alterations  throughout  the  model.  Additionally,  information 
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can  be  gathered  and  organized  using  a  spreadsheet  and  easily  transferred  into  Arena. 
An  example  of  such  a  spreadsheet  is  given  in  Appendix  D. 

The  validation  and  verification  effort  undertaken  during  and  after  model  de¬ 
velopment  lends  credibility  to  the  IPM.  Multiple  reviews  of  the  conceptual  model 
by  SMEs  helped  to  ensure  that  appropriate  assumptions  and  details  were  included. 
Furthermore,  multiple  detailed  walk-throughs  of  the  implemented  model  helped  to 
ensure  that  the  implementation  of  the  IPM  in  Arena  was  both  conceptually  and 
and  technically  correct.  The  use  of  a  notional  baseline  system  and  seven  case  studies 
to  stress  the  model  lends  additional  credibility  to  the  model  given  that  the  changes 
induced  for  the  seven  cases  resulted  in  expected  outcomes. 

In  spite  of  the  effort  undertaken  to  develop  a  complete,  valid  and  credible 
model,  two  limitations  still  remain.  The  first  limitation  is  with  the  detail  included  in 
the  Communications  submodel  of  the  IPM.  The  Communications  submodel  contains 
a  very  simple  representation  of  a  traditional  communications  environment.  That 
is,  it  only  provides  communications  delays  in  relatively  small  number  of  areas  and 
includes  minimal  communications  resources.  Although  such  details  are  desirable 
even  in  a  high  level  model,  various  constraints  prevented  their  inclusion  for  this 
study.  The  focus  taken  for  the  Communications  submodel  was  that  it  correctly  route 
entities  between  the  submodels  based  on  various  entity  attributes.  The  validation 
and  verihcation  effort  as  well  as  the  case  studies  confirmed  that  this  core  functionality 
of  the  Communications  submodel  works  correctly. 

The  second  limitation  involves  the  issue  of  information  integration  or  fusion. 
This  issue  was  intentionally  left  out  during  early  model  development.  Later  in  model 
development  there  was  some  indication  that  addressing  the  fusion  problem  would  be 
beneficial  for  some  studies.  The  result  was  a  minimal  fusion  model  which  required 
broad  assumptions  that  were  not  easily  integrated  with  the  rest  of  the  model.  At 
this  point  we  determined  further  development  of  the  fusion  capability  was  beyond 
the  scope  of  this  study.  In  order  to  appropriately  address  the  fusion  issue,  one  must 
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also  determine  a  way  to  model  the  breakdown  of  large  problems  into  the  “minimal” 
pieces  of  information  modeled  in  the  IPM.  Once  that  issue  is  addressed,  reassembling 
the  pieces  becomes  a  simpler  problem.  The  fusion  portions  of  the  model  were  not 
tested  and  are  not  recommended  for  use.  The  fusion  portions  are  easily  disabled, 
and  in  fact,  were  disabled  for  the  baseline  and  all  seven  case  studies. 

Overall,  the  intent  of  developing  a  generalized  framework  of  the  intelligence 
process  for  simulation  studies  has  been  successful.  The  addition  of  relevant  per¬ 
formance  measures,  such  as  proportion  of  requirements  met,  allows  the  IPM  to  be 
readily  used  for  many  simulation  studies.  Furthermore,  the  partitioning  of  those 
proportions  into  various  categories  allows  additional  insight  into  the  process. 

5.2  Application 

Given  that  an  analyst  performing  simulation  studies  using  the  IPM  under¬ 
stands  the  limitations  of  the  Communications  submodel  and  how  those  limitations 
might  impact  simulation  results,  they  could  immediately  begin  using  the  IPM  for 
simulation  studies.  One  potential  area  of  application  for  the  IPM  is  to  examine  the 
TPED,  TPPU,  and  other  hybrid  variations  of  the  intelligence  process  in  a  “satu¬ 
rated”  intelligence  environment.  In  other  words,  rather  than  examine  each  of  the 
variations  independent  of  each  other,  replace  a  portion  of  the  current  process  with 
that  variation  and  examine  the  impact  on  system  performance.  Another  potential 
area  of  application  for  the  IPM  is  to  examine  the  impact  of  various  proposed  changes 
to  collection  or  other  resources.  Studies  such  as  these  could  lend  insight  into  which 
suggested  changes  might  be  more  benehcial. 

5.3  Future  Research 

Aside  from  the  various  studies  that  can  be  conducted  using  the  model,  the 
primary  areas  of  future  research  involve  modifying  the  model  structure.  The  two 
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top  candidates  for  additional  research  relate  to  the  two  concerns  that  were  recapped 
in  Section  5.1.  The  top  candidate  is  the  development  of  a  more  robust  and  realistic 
communications  model.  Such  a  communications  model  would  need  to  maintain  the 
functionality  of  the  current  Communications  submodel  which  appropriately  routes 
entities  between  other  submodels  while  increasing  the  realism  and  robustness.  The 
challenge  for  this  research  area  is  selecting  an  appropriate  level  of  detail  while  main¬ 
taining  the  generalized  nature  of  the  IPM.  The  second  candidate  for  additional  re¬ 
search  is  an  investigation  into  how  high  level  problems  are  decomposed  into  smaller, 
“bite-sized”  information  requests  that  are  currently  handled  by  the  IPM.  In  concert 
with  such  an  investigation  would  be  the  development  and  addition  of  such  a  break¬ 
down  model  as  well  as  the  development  and  addition  of  fusion  within  the  model 
to  reassemble  information  into  the  original  requests.  Other  areas  of  research  could 
involve  adding  additional  detail  to  any  particular  submodel.  However,  care  should 
be  taken  to  not  include  too  much  detail,  as  it  would  defeat  the  purpose  of  a  high 
level  model  to  be  used  for  quick  look  studies.  Furthermore,  additional  details  in 
these  submodels  may  be  of  marginal  beneht  when  compared  to  research  into  the 
communications  model.  The  addition  of  these  areas  of  study  to  the  model  could 
greatly  broaden  the  scope  of  problems  the  simulation  could  be  used  to  examine  and 
allow  additional  insight  into  the  intelligence  process. 
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Appendix  A.  List  of  Abbreviations  and  Acronyms 

BL  Baseline 

C4ISR  Command,  Control,  Communications,  Computers,  Intelligence,  Surveil¬ 
lance,  and  Reconnaissance 

Cl  Counterintelligence 

CONOPS  Concept  of  operations 

COSMOS  C4ISR  Space  and  Missile  Operations  Simulator 

DES  Discrete  event  simulation 

HUMINT  Human  Intelligence 

IMINT  Imagery  Intelligence 

INS  Information  needs  satisfaction 

IPM  Intelligence  Process  Model 

ISR  Intelligence,  surveillance,  and  reconnaissance 

ISR-PET  ISR  Performance  Evaluation  Tool 

ISRSIM  Intelligence,  Surveillance  and  Reconnaissance  Simulator 

LVF  Lowest  value  first 

MASINT  Measurement  and  Signature  Intelligence 
NIS  Number  in  system  (see  WIP) 

NQ  Number  in  queue 

NSS  National  Security  Strategy 

NSSA  National  Security  Space  Architect 

OSINT  Open-Source  Intelligence 

PIR  Priority  Information  Request 
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QQT  Quality,  quantity,  and  timeliness 

QQTI  Qnality,  quantity,  timeliness,  and  information  needs  satisfaction 

QUICM  Quick  ISR  CONOPS  Modeler 

RFI  Request  for  information 

SAR  Synthetic  Apertnre  Radar 

SIGINT  Signals  Intelligence 

SME  Snbject  Matter  Expert 

TECHINT  Technical  Intelligence 

TIS  Time  in  system 

TPED  Tasking,  processing,  exploitation,  dissemination 

TPPU  Tasking,  processing,  posting,  using 

V&V  Validation  and  verihcation 

VBA  Visnal  Basic  for  Applications 

WIP  Work  in  process 

WT  Wait  time 
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Appendix  B.  Exported  VBA  Class  File 

The  following  VBA  code  was  exported  from  the  Intelligence  Process  Model  version 
2.7.  It  contains  code  for  six  VBA  blocks  used  in  the  Communications  submodel. 

VERSION  1.0  CLASS 
BEGIN 

MultiUse  =  -1  ’True 
END 

Attribute  VB_Naine  =  "ThisDocument" 

Attribute  VB_Creatable  =  False 
Attribute  VB_PredeclaredId  =  True 
Attribute  VB_Exposed  =  True 
Private  Sub  VBA_Block_l_Fire  () 

’  Determine  NextStation  after  leaving  Planning 

Dim  m  As  Model 

Dim  s  As  SIMAN 

Set  m  =  ThisDocument .Model 

Set  s  =  m. SIMAN 

’variable  to  hold  next  station 

Dim  Next_Station  As  Double 

’Dim  variables  to  hold  station  variables  locally  for  easier  access 

Dim  Stn_Planning  As  Double 

Dim  Stn_Library  As  Double 

Dim  Stn_Collection  As  Double 

Dim  Stn_Processing  As  Double 

Dim  Stn_Exploitation  As  Double 

Dim  Stn_Analysis  As  Double 

Dim  Stn_Production  As  Double 

Dim  Stn_Dissemination  As  Double 

Dim  Stn_Integration  As  Double 

Dim  Stn_Evaluation  As  Double 

’Dim  variables  to  get  entity  processing  steps 

Dim  Need.Collect  As  Double 

Dim  Need-Process  As  Double 

Dim  Need-Exploit  As  Double 

Dim  Need-Analyze  As  Double 

Dim  Need-Produce  As  Double 

Dim  Need-Disseminate  As  Double 

Dim  Need-Integrate  As  Double 

’Assign  local  variables  values  of  stations 

Stn-PIanning  =  s . VariableArrayValue (s . SymboINumber ("PlanningStation") ) 
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Stn_Library  =  s . VariableArrayValue(s . SymbolNumber ("LibraryStation") ) 

Stn_Collection  =  s .VariableArrayValue(s .SymbolNumber ("CollectionStation") ) 
Stn_Processing  =  s .VariableArrayValue(s .SymbolNumber ("ProcessingStation") ) 
Stn_Exploitatioii  =  s . VariableArrayValue (s . SymbolNumber ( "Exploitat ionStat ion") ) 
Stn_Analysis  =  s .VariableArrayValue (s . SymbolNumber ("AnalysisStation") ) 
Stn_Production  =  s . VariableArrayValue (s . SymbolNumber ("ProductionStation") ) 
Stn_Dissemination  =  s . VariableArrayValue (s .SymbolNumber ("DisseminationStation") ) 
Stn_Integration  =  s . VariableArrayValue (s .SymbolNumber ("IntegrationStation")) 
Stn_Evaluation  =  s . VariableArrayValue (s .SymbolNumber ("EvaluationStation") ) 

’Assign  local  variables  values  of  stations  required  for  processing 
Need.Collect  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Collect")) 
Need_Process  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Process")) 
Need-Exploit  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Exploit")) 
Need_Analyze  =  s .EntityAttribute(s. ActiveEntity ,  s .SymbolNumber ("Analyze") ) 
Need-Produce  =  s .EntityAttribute(s. ActiveEntity ,  s .SymbolNumber ("Produce") ) 
Need-Disseminate  =  s .EntityAttribute(s .ActiveEntity ,  s. SymbolNumber ("Disseminate") ) 
Need-Integrate  =  s . Ent ityAttribute ( s . ActiveEntity ,  s . SymbolNumber ( " Integrate " ) ) 
’Determine  where  to  go  next 
If  Need-Collect  =  1  Then 

Next-Station  =  Stn-Collection 
Elself  Need-Process  =  1  Then 

Next-Station  =  Stn-Processing 
Elself  Need-Exploit  =  1  Then 

Next-Station  =  Stn-Exploitation 
Elself  Need-Analyze  =  1  Then 

Next-Station  =  Stn-Analysis 
Elself  Need-Produce  =  1  Then 

Next-Station  =  Stn-Production 
Elself  Need-Disseminate  =  1  Then 

Next-Station  =  Stn-Dissemination 
Elself  Need-Integrate  =  1  Then 

Next-Station  =  Stn-Integration 

Else 

Next-Station  =  Stn-Evaluation 
End  If 

’Assign  Entity .NextStation  to  Next-Station 

s .Ent ityAttribute (s .ActiveEntity ,  s .SymbolNumber ("NextStation"))  =  Next -Station 
End  Sub 

Private  Sub  VBA_Block_2_Fire  () 

’  Determine  NextStation  after  leaving  Collection 

Dim  m  As  Model 

Dim  s  As  SIMAN 

Set  m  =  ThisDocument .Model 
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Set  s  =  m.SIMAN 


’variable  to  hold  next  station 
Dim  Next_Station  As  Double 

’Dim  variables  to  hold  station  variables  locally  for  easier  access 

Dim  Stn_Planning  As  Double 

Dim  Stn_Library  As  Double 

Dim  Stn_Collection  As  Double 

Dim  Stn_Processing  As  Double 

Dim  Stn_Exploitation  As  Double 

Dim  Stn_Analysis  As  Double 

Dim  Stn_Production  As  Double 

Dim  Stn_Dissemination  As  Double 

Dim  Stn.Integration  As  Double 

Dim  Stn_Evaluation  As  Double 

’Dim  variables  to  get  entity  processing  steps 

Dim  Need.Collect  As  Double 

Dim  Need_Process  As  Double 

Dim  Need_Exploit  As  Double 

Dim  Need_Analyze  As  Double 

Dim  Need-Produce  As  Double 

Dim  Need_Disseminate  As  Double 

Dim  Need-Integrate  As  Double 

’Assign  local  variables  values  of  stations 

Stn_Planning  =  s . VariableArrayValue (s . SymbolNumber ("PlanningStation") ) 

Stn_Library  =  s . VariableArrayValue ( s . SymbolNumber ( "LibraryStat ion" ) ) 

Stn.Collection  =  s . VariableArrayValue (s .SymbolNumber ("CollectionStation") ) 
Stn_Processing  =  s . VariableArrayValue (s .SymbolNumber ("ProcessingStation") ) 
Stn_Exploitation  =  s .VariableArrayValue (s . SymbolNumber ( "Exploitat ionStat ion") ) 
Stn_Analysis  =  s .VariableArrayValue (s . SymbolNumber (" AnalysisSt at ion") ) 
Stn_Production  =  s . VariableArrayValue (s . SymbolNumber ("ProductionStation") ) 
Stn_Dissemination  =  s . VariableArrayValue (s .SymbolNumber ("DisseminationStation") ) 
Stn.Integration  =  s . VariableArrayValue (s .SymbolNumber ("IntegrationStation")) 
Stn_Evaluation  =  s . VariableArrayValue (s .SymbolNumber ("EvaluationStation") ) 

’Assign  local  variables  values  of  stations  required  for  processing 
Need-Collect  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Collect")) 
Need-Process  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Process")) 
Need-Exploit  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Exploit")) 
Need-Analyze  =  s .EntityAttribute(s. ActiveEntity ,  s .SymbolNumber ("Analyze") ) 
Need-Produce  =  s .EntityAttribute(s. ActiveEntity ,  s .SymbolNumber ("Produce") ) 
Need-Disseminate  =  s .EntityAttribute(s .ActiveEntity ,  s. SymbolNumber ("Disseminate") ) 
Need-Integrate  =  s . Ent ityAttribute ( s . ActiveEntity ,  s . SymbolNumber ( " Integrate " ) ) 
’Determine  where  to  go  next 
If  Need-Process  =  1  Then 
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Next_Statioii  =  Stn_Processing 
Elself  Need_Exploit  =  1  Then 

Next_Station  =  Stn_Exploitation 
Elself  Need_Analyze  =  1  Then 

Next_Station  =  Stn_Analysis 
Elself  Need_Produce  =  1  Then 

Next_Station  =  Stn_Production 
Elself  Need-Disseminate  =  1  Then 

Next_Station  =  Stn_Disseinination 
Elself  Need.Integrate  =  1  Then 

Next_Station  =  Stn.Integration 

Else 

Next-Station  =  Stn_Evaluation 
End  If 

’Assign  Entity .NextStation  to  Next_Station 

s .EntityAttribute(s . ActiveEntity,  s .SymbolNumberC "NextStation"))  =  Next  .Station 
End  Sub 

Private  Sub  VBA_Block_3_Fire  () 

’  Determine  NextStation  after  leaving  Processing 

Dim  m  As  Model 

Dim  s  As  SIMAN 

Set  m  =  ThisDocument .Model 

Set  s  =  m. SIMAN 

’variable  to  hold  next  station 

Dim  Next-Station  As  Double 

’Dim  variables  to  hold  station  variables  locally  for  easier  access 

Dim  Stn_Planning  As  Double 

Dim  Stn_Library  As  Double 

Dim  Stn.Collection  As  Double 

Dim  Stn.Processing  As  Double 

Dim  Stn.Exploitation  As  Double 

Dim  Stn.Analysis  As  Double 

Dim  Stn.Production  As  Double 

Dim  Stn_Dissemination  As  Double 

Dim  Stn.Integration  As  Double 

Dim  Stn.Evaluation  As  Double 

’Dim  variables  to  get  entity  processing  steps 

Dim  Need-Collect  As  Double 

Dim  Need-Process  As  Double 

Dim  Need-Exploit  As  Double 

Dim  Need-Analyze  As  Double 

Dim  Need-Produce  As  Double 

Dim  Need-Disseminate  As  Double 
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Dim  Need_Integrate  As  Double 

’Assign  local  variables  values  of  stations 

Stn_Planning  =  s . VariableArrayValue (s . SymbolNumber ("PlanningStation") ) 

Stn_Library  =  s . VariableArrayValue ( s . SymbolNumber ( "LibrarySt at ion" ) ) 

Stn.Collection  =  s . VariableArrayValue (s .SymbolNumber ("CollectionStation") ) 
Stn_Processing  =  s . VariableArrayValue (s .SymbolNumber ("ProcessingStation") ) 
Stn_Exploitation  =  s .VariableArrayValue (s . SymbolNumber ( "Exploitat ionStat ion") ) 
Stn_Analysis  =  s .VariableArrayValue (s . SymbolNumber (" AnalysisSt at ion") ) 
Stn_Production  =  s . VariableArrayValue (s . SymbolNumber ("ProductionStation") ) 
Stn_Dissemination  =  s . VariableArrayValue (s .SymbolNumber ("DisseminationStation") ) 
Stn.Integration  =  s . VariableArrayValue (s .SymbolNumber ("IntegrationStation")) 
Stn_Evaluation  =  s . VariableArrayValue (s .SymbolNumber ("EvaluationStation") ) 

’Assign  local  variables  values  of  stations  required  for  processing 
Need_Collect  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Collect")) 
Need_Process  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Process")) 
Need_Exploit  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Exploit")) 
Need_Analyze  =  s .EntityAttribute(s. ActiveEntity ,  s .SymbolNumber ("Analyze") ) 
Need_Produce  =  s . EntityAttribute (s . ActiveEntity ,  s . SymbolNumber ( "Produce " ) ) 
Need_Disseminate  =  s . EntityAttribute (s .ActiveEntity ,  s. SymbolNumber ("Disseminate") ) 
Need-Integrate  =  s . EntityAttribute ( s . ActiveEntity ,  s . SymbolNumber ( " Integrate " ) ) 
’Determine  where  to  go  next 
If  Need_Exploit  =  1  Then 

Next-Station  =  Stn-Exploitation 
Elself  Need-Analyze  =  1  Then 

Next-Station  =  Stn-Analysis 
Elself  Need-Produce  =  1  Then 

Next-Station  =  Stn_Production 
Elself  Need-Disseminate  =  1  Then 

Next-Station  =  Stn-Dissemination 
Elself  Need-Integrate  =  1  Then 

Next-Station  =  Stn-Integration 

Else 

Next-Station  =  Stn-Evaluation 
End  If 

’Assign  Entity .NextStation  to  Next-Station 

s . EntityAttribute (s .ActiveEntity,  s .SymbolNumber ("NextStation"))  =  Next -Station 
End  Sub 

Private  Sub  VBA-Block-4-Fire  () 

’  Determine  NextStation  after  leaving  Exploitation 

Dim  m  As  Model 

Dim  s  As  SIMAN 

Set  m  =  ThisDocument .Model 

Set  s  =  m. SIMAN 
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variable  to  hold  next  station 


Dim  Next_Station  As  Double 

’Dim  variables  to  hold  station  variables  locally  for  easier  access 

Dim  Stn_Planning  As  Double 

Dim  Stn_Library  As  Double 

Dim  Stn.Collection  As  Double 

Dim  Stn_Processing  As  Double 

Dim  Stn_Exploitation  As  Double 

Dim  Stn_Analysis  As  Double 

Dim  Stn_Production  As  Double 

Dim  Stn_Dissemination  As  Double 

Dim  Stn.Integration  As  Double 

Dim  Stn_Evaluation  As  Double 

’Dim  variables  to  get  entity  processing  steps 

Dim  Need_Collect  As  Double 

Dim  Need_Process  As  Double 

Dim  Need_Exploit  As  Double 

Dim  Need_Analyze  As  Double 

Dim  Need_Produce  As  Double 

Dim  Need_Disseminate  As  Double 

Dim  Need_Integrate  As  Double 

’Assign  local  variables  values  of  stations 

Stn_Planning  =  s . VariableArrayValue (s . SymbolNumber ("PlanningStation") ) 

Stn_Library  =  s . VariableArrayValue (s . SymbolNumber ("LibraryStat ion") ) 

Stn.Collection  =  s . VariableArrayValue (s .SymbolNumber ("CollectionStation") ) 
StnJrocessing  =  s . VariableArrayValue (s .SymbolNumber ("ProcessingStation") ) 
StnJlxploitation  =  s .VariableArrayValue (s . SymbolNumber ( "Exploitat ionStat ion") ) 
Stn_Analysis  =  s .VariableArrayValue (s . SymbolNumber (" AnalysisSt at ion") ) 
StnJ’roduction  =  s . VariableArrayValue (s .SymbolNumber ("ProductionStation") ) 
StnJ)issemination  =  s . VariableArrayValue (s .SymbolNumber ("DisseminationStation") ) 
Stn.Integration  =  s . VariableArrayValue (s .SymbolNumber ("IntegrationStation")) 
StnJlvaluation  =  s . VariableArrayValue (s .SymbolNumber ("EvaluationStation") ) 

’Assign  local  variables  values  of  stations  required  for  processing 
Need.Collect  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Collect")) 
Need_Process  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Process")) 
Need.Exploit  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Exploit")) 
Need.Analyze  =  s .EntityAttribute(s. ActiveEntity ,  s .SymbolNumber ("Analyze") ) 
Need.Produce  =  s .EntityAttribute(s. ActiveEntity ,  s .SymbolNumber ("Produce") ) 
Need.Disseminate  =  s .EntityAttribute(s .ActiveEntity ,  s. SymbolNumber ("Disseminate") ) 
Need.Int egrate  =  s . Ent ityAttribute ( s . ActiveEntity ,  s . SymbolNumber ( " Integrate " ) ) 
’Determine  where  to  go  next 
If  Need.Analyze  =  1  Then 

Next.Station  =  Stn.Analysis 
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Elself  Need_Produce  =  1  Then 


Next_Station  =  Stn_Production 
Elself  Need_Disseininate  =  1  Then 

Next_Station  =  Stn_Disseinination 
Elself  Need.Integrate  =  1  Then 

Next_Station  =  Stn.Integration 

Else 

Next_Station  =  Stn_Evaluation 
End  If 

’Assign  Entity .NextStation  to  Next.Station 

s .EntityAttribute(s . ActiveEntity,  s .SymbolNumberC "NextStation"))  =  Next.Station 
End  Sub 

Private  Sub  VBA_Block_5 J^ire  () 

’  Determine  NextStation  after  leaving  Analysis 

Dim  m  As  Model 

Dim  s  As  SIMAN 

Set  m  =  ThisDocument .Model 

Set  s  =  m. SIMAN 

’variable  to  hold  next  station 

Dim  Next-Station  As  Double 

’Dim  variables  to  hold  station  variables  locally  for  easier  access 

Dim  Stn_Planning  As  Double 

Dim  Stn_Library  As  Double 

Dim  Stn.Collection  As  Double 

Dim  Stn_Processing  As  Double 

Dim  Stn_Exploitation  As  Double 

Dim  Stn_Analysis  As  Double 

Dim  Stn_Production  As  Double 

Dim  Stn_Dissemination  As  Double 

Dim  Stn.Integration  As  Double 

Dim  Stn_Evaluation  As  Double 

’Dim  variables  to  get  entity  processing  steps 

Dim  Need-Collect  As  Double 

Dim  Need_Process  As  Double 

Dim  Need_Exploit  As  Double 

Dim  Need-Analyze  As  Double 

Dim  Need-Produce  As  Double 

Dim  Need-Disseminate  As  Double 

Dim  Need-Integrate  As  Double 

’Assign  local  variables  values  of  stations 

Stn_Planning  =  s . VariableArrayValue (s . SymbolNumber ("PlanningStation") ) 
Stn-Library  =  s . VariableArrayValue ( s . SymbolNumber ( "LibrarySt at ion" ) ) 
Stn.Collection  =  s . VariableArrayValue (s .SymbolNumber ("CollectionStation") ) 
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Stn_Processiiig  =  s . VariableArrayValue(s .SymbolNumber ("ProcessingStation") ) 
Stn_Exploitatioii  =  s . VariableArrayValue (s . SymbolNumber ( "Exploitat ionStat ion") ) 
Stn_Analysis  =  s .VariableArrayValue (s . SymbolNumber ("AnalysisStation") ) 
Stn_Production  =  s . VariableArrayValue (s . SymbolNumber ("ProductionStation") ) 
Stn_Dissemination  =  s . VariableArrayValue (s .SymbolNumber ("DisseminationStation") ) 
Stn.Integration  =  s . VariableArrayValue (s .SymbolNumber ("IntegrationStation")) 
Stn_Evaluation  =  s . VariableArrayValue (s .SymbolNumber ("EvaluationStation") ) 

’Assign  local  variables  values  of  stations  required  for  processing 
Need_Collect  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Collect")) 
Need_Process  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Process")) 
Need_Exploit  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Exploit")) 
Need_Analyze  =  s .EntityAttribute(s. ActiveEntity ,  s .SymbolNumber ("Analyze") ) 
Need-Produce  =  s .EntityAttribute(s. ActiveEntity ,  s .SymbolNumber ("Produce") ) 
Need_Disseminate  =  s .EntityAttribute(s .ActiveEntity ,  s. SymbolNumber ("Disseminate") ) 
Need-Integrate  =  s . Ent ityAttribute ( s . ActiveEntity ,  s . SymbolNumber ( " Integrate " ) ) 
’Determine  where  to  go  next 
If  Need-Produce  =  1  Then 

Next-Station  =  Stn-Production 
Elself  Need-Disseminate  =  1  Then 

Next-Station  =  Stn-Dissemination 
Elself  Need-Integrate  =  1  Then 

Next-Station  =  Stn-Integration 

Else 

Next-Station  =  Stn-Evaluation 
End  If 

’Assign  Entity .NextStation  to  Next-Station 

s .Ent ityAttribute (s .ActiveEntity,  s .SymbolNumber ("NextStation"))  =  Next -Station 
End  Sub 

Private  Sub  VBA_Block_6_Fire  () 

’  Determine  NextStation  after  leaving  Production 

Dim  m  As  Model 

Dim  s  As  SIMAN 

Set  m  =  ThisDocument .Model 

Set  s  =  m. SIMAN 

’variable  to  hold  next  station 

Dim  Next-Station  As  Double 

’Dim  variables  to  hold  station  variables  locally  for  easier  access 

Dim  Stn-Planning  As  Double 

Dim  Stn-Library  As  Double 

Dim  Stn-Collection  As  Double 

Dim  Stn-Processing  As  Double 

Dim  Stn-Exploitation  As  Double 

Dim  Stn-Analysis  As  Double 


B-8 


Dim  Stn_Production  As  Double 


Dim  Stn_Dissemination  As  Double 

Dim  Stn.Integration  As  Double 

Dim  Stn_Evaluation  As  Double 

’Dim  variables  to  get  entity  processing  steps 

Dim  Need.Collect  As  Double 

Dim  Need_Process  As  Double 

Dim  Need_Exploit  As  Double 

Dim  Need_Analyze  As  Double 

Dim  Need_Produce  As  Double 

Dim  Need_Disseminate  As  Double 

Dim  Need.Integrate  As  Double 

’Assign  local  variables  values  of  stations 

Stn_Planning  =  s . VariableArrayValue (s . SymbolNumber ("PlanningStation") ) 

Stn_Library  =  s . VariableArrayValue (s . SymbolNumber ("LibraryStat ion") ) 

Stn.Collection  =  s . VariableArrayValue (s .SymbolNumber ("CollectionStation") ) 
Stn_Processing  =  s . VariableArrayValue (s .SymbolNumber ("ProcessingStation") ) 
Stn_Exploitation  =  s .VariableArrayValue (s . SymbolNumber ( "Exploitat ionStat ion") ) 
Stn_Analysis  =  s .VariableArrayValue (s . SymbolNumber (" AnalysisSt at ion") ) 
Stn_Production  =  s . VariableArrayValue (s . SymbolNumber ("ProductionStation") ) 
Stn_Dissemination  =  s . VariableArrayValue (s .SymbolNumber ("DisseminationStation") ) 
Stn_Integration  =  s . VariableArrayValue (s .SymbolNumber ("IntegrationStation")) 
Stn_Evaluation  =  s . VariableArrayValue (s .SymbolNumber ("EvaluationStation") ) 

’Assign  local  variables  values  of  stations  required  for  processing 
Need.Collect  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Collect")) 
Need-Process  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Process")) 
Need-Exploit  =  s.EntityAttribute(s.ActiveEntity,  s. SymbolNumber ("Exploit")) 
Need-Analyze  =  s .EntityAttribute(s. ActiveEntity ,  s .SymbolNumber ("Analyze") ) 
Need-Produce  =  s .EntityAttribute(s. ActiveEntity ,  s .SymbolNumber ("Produce") ) 
Need-Disseminate  =  s .EntityAttribute(s .ActiveEntity ,  s. SymbolNumber ("Disseminate") ) 
Need-Integrate  =  s . Ent ityAttribute ( s . ActiveEntity ,  s . SymbolNumber ( " Integrate " ) ) 
’Determine  where  to  go  next 
If  Need-Disseminate  =  1  Then 

Next-Station  =  Stn_Dissemination 
Elself  Need-Integrate  =  1  Then 

Next-Station  =  Stn-Integration 

Else 

Next-Station  =  Stn-Evaluation 
End  If 

’Assign  Entity .NextStation  to  Next-Station 

s .Ent ityAttribute (s .ActiveEntity,  s .SymbolNumber ("NextStation"))  =  Next -Station 
End  Sub 
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Appendix  C.  Model  Flow  Charts 

The  following  figures  contain  flow  charts  used  for  model  validation  and  verihcation. 
The  hrst  flow  char  depicts  the  potential  flow  of  entities  due  to  the  modularity  of  the 
model.  The  current  implementation  is  similar  to  the  planned  v2.5  implementation 
except  that  the  Planning,  Library  Search,  and  Evaluation  portions  have  been  fully 
modularized.  All  possible  routes  depicted  are  not  realized  because  of  the  core  model 
logic  within  the  Communications  submodel.  The  core  logic  of  the  current  v2.7 
model  is  essentially  the  same  as  the  planned  v2.5  model  except  that  some  RFIs  do 
not  undergo  a  library  search  by  default.  Refer  to  Chapter  3  for  details  on  the  current 
model  implementation. 
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Intelligence  Process  Model  v2.0 


Intelligence  Process  Modei  v2.5 
Entity  Fiow  (pianned) 


This  link  may  or  may  not  be  implemented. 
Q:  Once  an  item  is  located,  is  a  significant 
usage  of  comms  needed  to  get  the  item? 
Maybe  in  the  case  of  imagery,  but  it  may 
make  more  sense  to  include  this  usage 
between  library  search  and  the  next 
phase.  May  also  be  needed  if  model  logic 
needs  to  be  altered. 


Figure  C.l  Charts  depicting  possible  entity  flow  in  earlier  model  versions. 
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Intelligence  Process  Model  v2.0 
Flowchart  of  basic  model  logic 


1  Dec  2003 


Figure  C.2  Flow  chart  of  the  IPM  v2.0  model  logic. 
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Intelligence  Process  Model  v2.5 
Flowchart  of  basic  model  logic  (planned) 


1  Dec  2003 


Figure  C.3  Flow  chart  of  the  IPM  v2.5  model  logic. 
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Appendix  D.  Data  Request  Sheet  with  Data  for 

Sample  Study 

The  following  figures  contain  a  copy  of  the  data  request  sheet  used  for  to  collect  data 
for  the  sample  study.  It  includes  an  exhaustive  list  of  the  data  required  for  expected 
studies.  The  first  page  of  the  request  sheet  gives  some  basic  information  about  the 
types  of  information  expected  for  various  attributes  assigned  in  the  remainder  of 
the  request  sheet.  Additional  information  about  syntax  and  parameters  of  Arena 
probability  distributions  and  expressions  can  be  found  in  the  software  help. 
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Fixed  Values 

Name  Value 

Max  #  of  Users  (User)  Max#U  =  5 

Max  #  of  Info  Sources  (InfoSource)  Max#IS  =  1 3 
Levels  of  Quality  (QuaIR)  #QLR  =  5 

Levels  of  Quality  (QualA)  #QLA  =  6 


Comments 

Hard  coded  into  model.  Can  be  changed,  but  requires  significant  effort.  This  impacts  any 
portion  of  the  model  that  is  user  dependent. 

Hard  coded  into  model.  Can  be  changed,  but  requires  significant  effort.  This  impacts  any 
portion  of  the  model  that  is  dependent  on  #IS. 

Hard  coded  into  model.  Can  be  changed,  but  requires  significant  effort.  This  impacts  any 
portion  of  the  model  that  is  dependent  on  #QLR. 

Hard  coded  into  model.  Can  be  changed,  but  requires  significant  effort.  This  impacts  any 
portion  of  the  model  that  is  dependent  on  #QLA. 


Base  Time  Unit 


Set  to  Hours  as  a  default.  Could  be  changed  to  Days,  Hours,  Minutes,  or  Seconds,  but  ensure 
this  change  is  made  everywhere  (i.e.,  in  all  delay,  process,  create,  etc.  blocks) . 


Mlsc  Notes 

All  Arrays  must  be  completely  populated.  However,  any  entries  beyond  #U  and  #IS  can  be  populated  with  zeros. 
Quality  levels  range  from  0  to  5  with  5  being  the  best  quality 
Required  Quality  must  have  integer  values  from  1  to  5 

Achieved  Quality  must  be  >=  0  {can  be  non-integral)  but  will  be  ranked  from  0  to  5  for  some  actions  in  the  model 
Logical  expressions  (i.e.,  A  ==  B  or  A>=  B)  evaluate  to  0  if  false  or  1  if  true 


The  information  sources  are  given  generically,  but  are  intended  to  be  used  at  a  high  aggregate  level.  Some  samples  are  given  below. 


InfoSources 

Source_01 
Source_02 
Source_03 
Source_04 
Source_05 
Source_06 
Source_07 
Source_08 
Source_09 
Source_10 
Source_1 1 
Source_12 
Source_13 


Sample  Source 
IMINT 
HUMINT 
SIGINT 
MAS  I  NT 
OSINT 
TECHINT 
Cl 

SBR 
MQVINT 
Radar 
Surveil 
Recon 
Coalition 


Attributes  of  entites 

Attribute  Name 

User 

Standard 

InfoSource 

TimeR 

QuaIR 

Priority_User 

Collect 

Process 

Exploit 

Analyze 

Produce 

Disseminate 

Integrate 


or  other  expressions  can  be  referenced  directly. 

Values  Comments 

integer  1  -5  Denotes  user  where  item  originated 

integer  0-1  Denotes  whether  an  item  is  a  standing/standard  request  (1)  or  an  additional  request  (0) 
integer  1  -1 3  Denotes  the  information  source  required  for  the  item 
real  >  0  Denotes  time  from  creation  of  item  when  it  is  required 

integer  1-5  Denotes  required  quality  of  item  (5  is  highest  quality) 

integer  1-5  Denotes  user  priority  of  an  item  (1  is  highest  priority) 

integer  0-1  Denotes  if  collection  should  occur  (1)  or  not  (0)  for  this  item 

integer  0-1  Denotes  if  processing  should  occur  (1 )  or  not  (0)  for  this  item 

integer  0-1  Denotes  if  exploitation  should  occur  (1 )  or  not  (0)  for  this  item 

integer  0-1  Denotes  if  analysis  should  occur  (1 )  or  not  (0)  for  this  item 

integer  0-1  Denotes  if  production  should  occur  (1 )  or  not  (0)  for  this  item 

integer  0-1  Denotes  if  dissemination  should  occur  (1 )  or  not  (0)  for  this  item 

integer  0-1  Denotes  if  user  integration  should  occur  (1 )  or  not  (0)  for  this  item 


Expression  Name 
Timely 

QualMet 

AgeMet 

QualARank 


Values 

integer  0-1 

integer  0-1 
integer  0-1 
integer  0-1 


Expression  Comments 
(  TNQW  -  Entity.CreateTime  )  -  TimeR  <=  0 
Evaluates  if  an  item  is  timely  or  not 
QualA  >=  QuaIR 

Evaluates  if  an  item  meets  the  quality  requirement 
AgeMetInLibrary 

Evaluates  if  the  age  requirement  was  met  during  a  library  search 
MN(  MX(  ANINT(QualA)  ,0)  +  ^  ,  6  ) 

Ranks  actual/achieved  quality  into  5  levels  1-6  (corresponds  to  real  levels  0-5) 
Functions:  MN  =  min,  MX  =  max,  ANINT  =  round  to  nearest  integer 
ANINT  could  be  replaced  by  AINT  (truncate)  if  it  makes  more  sense 


Supported  Probability  Distributions 

Beta 

Empirical  Continuous 

Empirical  Discrete 

k-Erlang 

Exponential 

Gamma 

Johnson 

Weibull 

Lognormal 

Normal 

NSExpo 

Poisson 

Triangular 

Uniform 

Supported  Mathmatical  Expressions 

Mathmatical  expressions  other  than  probability  distributions  can  also  be  used  in  lieu  of  or  in  conjuction  with  probablility  distributions  if  desired. 
Contents  of  package  (pdf)  may  need  to  be  activated  2-3  times  before  it  finds  Acrobat. 
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Figure  D.2 


Data  Request  Sheet  Page  1 
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Figure  D.3  Data  Request  Sheet  Page  2 
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Figure  D.4  Data  Request  Sheet  Page  3 
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InfoAnalyzeQualA  Each  entry  of  the  array  corresponds  to  the  distribution  of  quality  achieved  for  information  from  a  given  information  source  as  altered  by  analysis  by  an  all  source  analyst  with  a  given  specialty. 

Inclusion  of  QualA  in  these  expressions  is  critical,  otherwise  the  previous  quality  will  be  completely  overwritten. 
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Data  Request  Sheet  Page  8 
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Figure  D.IO 


Data  Request  Sheet  Page  9 
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Figure  D.ll 
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Figure  D.12 
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Figure  D.13 


Data  Request  Sheet  Page  12 
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Figure  D.14 


Data  Request  Sheet  Page  13 
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Data  Request  Sheet  Page  14 
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Figure  D.16 


Data  Request  Sheet  Page  15 
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CommsQualA  Distribution  of  achieved  quality  as  it  is  affected  by  communications  as  items  are  transferred  between  steps  of  the  intelligence  process. 

Inclusion  of  QualA  in  these  expressions  is  critical,  otherwise  the  previous  quality  will  be  completely  overwritten. 
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I  CommsPostProcessTimes  Distribution  of  time  taken  to  post  information  from  a  given  source  to  centralized  repositories  after  processing 


Figure  D.18 


Data  Request  Sheet  Page  17 
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Figure  D.19 


Data  Request  Sheet  Page  18 
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Appendix  E.  Implemented  Statistics 

The  following  table  is  a  list  of  all  the  statistics  that  were  created  to  gather  information 
about  the  system.  Note  that  this  list  does  not  include  any  of  the  built  in  statistics 
such  as  resource  utilization  or  wait  times  in  queues.  These  statistics  are  dehned 
using  the  data  collected  in  the  variables  given  in  Table  3.4. 


Statistic 

Definition 

Pct_Timely_Pr_l 

S_Num_Timely_Priority(l)  /  S_Num_Out_Priority(l) 

Pct_Timely_Pr_2 

S_Num_Timely_Priority(2)  /  S_Num_Out_Priority(2) 

Pct_Timely_Pr_3 

S_Num_Timely_Priority(3)  /  S_Num_Out_Priority(3) 

Pct_Timely_Pr_4 

S_Num_Timely_Priority(4)  /  S_Num_Out_Priority(4) 

Pct_Timely_Pr_5 

S_Num_Timely_Priority(5)  /  S_Num_Out_Priority(5) 

Pet  .Timely  _U sr  _1 

S_Num_Timely_User(l)  /  S_Num_Out_User(l) 

Pct_Timely_Usr_2 

S_Num_Timely_User(2)  /  S_Num_Out_User(2) 

Pct_Timely_Usr_3 

S_Num_Timely_User(3)  /  S_Num_Out_User(3) 

Pct_Timely_Usr_4 

S_Num_Timely_User(4)  /  S_Num_Out_User(4) 

Pct_Timely_Usr_5 

S_Num_Timely_User(5)  /  S_Num_Out_User(5) 

Pct_Timelyv\dd 

S_Num_Timely_Standard(l)  /  S_Num_Out_Standard(l) 

Pct.Timely.Std 

S_Num_Timely_Standard(2)  /  S_Num_Out_Standard(2) 

Pct_QualMet_Pr_l 

S_Num_QuaLPriority(l)  /  S_Num_Out_Priority(l) 

Pct_QualMet_Pr_2 

S_Num_QuaLPriority(2)  /  S_Num_Out_Priority(2) 

Pct_QualMet_Pr_3 

S_Num_Qual_Priority(3)  /  S_Num_Out_Priority(3) 

Pct_QualMet_Pr_4 

S_Num_Qual_Priority(4)  /  S_Num_Out_Priority(4) 

Pct_QualMet_Pr_5 

S_Num_Qual_Priority(5)  /  S_Num_Out_Priority(5) 

Pet  -QualMet  _U  sr  _1 

S_Num_Qual_User(l)  /  S_Num_Out_User(l) 

Pct_QualMet_Usr_2 

S_Num_Qual_User(2)  /  S_Num_Out_User(2) 

Pct_QualMet_Usr_3 

S_Num_Qual_User(3)  /  S_Num_Out_User(3) 
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Statistic 

Definition 

Pct_QualMet_Usr_4 

S_Num_QuaLUser(4)  /  S_Num_Out_User(4) 

Pet  _QualMet  _U  sr  _5 

S_Num_QuaLUser(5)  /  S_Num_Out_User(5) 

Pct_QualMet^dd 

S_Num_QuaLStandard(l)  /  S_Num_Out_Standard(l) 

Pct_QualMet_Std 

S_Num_QuaLStandard(2)  /  S_Num_Out_Standard(2) 

Pct_QT_Pr_l 

S_Num_QT_Priority(l)  /  S_Num_Out_Priority(l) 

Pct_QT_Pr_2 

S_Num_QT_Priority(2)  /  S_Num_Out_Priority(2) 

Pct_QT_Pr_3 
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Appendix  F.  Additional  Simulation  Results 

The  following  sections  include  additional  simulation  results  that  were  not  presented 
in  Chapter  4. 
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Figure  F.2  Baseline  simulation  results  part  2. 
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Figure  F.3  Baseline  simulation  results  part  3. 
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F.2  Case  1  Results 
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Figure  F.5  Case  1  simulation  results  part  2. 
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Figure  F.6  Case  1  simulation  results  part  3. 
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F.3  Case  2  Results 
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0.9998 

0.9685 

0.9683 

1  0.6713 

0.8451 

0.6593 

2 

0.9907 

0.9678 

0.9591 

2  0.7647 

0.8148 

0.7283 

3 

0.6694 

0.9449 

0.6476 

3  0.7120 

0.8436 

0.6985 

4 

0.2820 

0.5534 

0.2713 

4  0.7397 

0.8483 

0.7308 

5 

0.0662 

0.1825 

0.0631 

5  0.7938 

0.8060 

0.7458 

Half-Widths 

Priority 

Timely 

QualMet 

Both 

User 

Timely 

QualMet 

Both 

1 

0.000004 

0.000069 

0.000070 

1  0.000417 

0.000325 

0.000414 

2 

0.000121 

0.000066 

0.000126 

2  0.000523 

0.000277 

0.000494 

3 

0.001053 

0.000174 

0.001024 

3  0.000371 

0.000287 

0.000366 

4 

0.001205 

0.001573 

0.001170 

4  0.000358 

0.000308 

0.000354 

5 

0.000576 

0.001251 

0.000551 

5  0.000439 

0.000307 

0.000450 

Averages 

InfoSource 

Timely 

QualMet 

Both 

Timely 

QualMet 

Both 

1 

0.7561 

0.6953 

0.6312 

Additional 

0.8558 

0.9599 

0.8286 

2 

0.7833 

0.8595 

0.7833 

Standard 

0.6367 

0.7243 

0.6157 

3 

0.7064 

0.8696 

0.7064 

4 

0.7215 

0.8620 

0.7215 

Half-Widths 

InfoSource 

Timely 

QualMet 

Both 

Timely 

QualMet 

Both 

1 

0.000327 

0.000310 

0.000328 

Additional 

0.000418 

0.000069 

0.00041 1 

2 

0.000533 

0.000307 

0.000533 

Standard 

0.000473 

0.000546 

0.000463 

3 

0.000510 

0.000396 

0.000510 

4 

0.000408 

0.000312 

0.000408 

Figure 

F.7  1 

Case  2  i 

simulation  results  part  1. 

F-5 


WIP 

Averages 


Average 

Max 

1689  max(max) 

RFI 

507.34 

1293.56 

1022  min{max) 

Half-Widths 

Average 

Max 

RFI 

1.6942 

13.3594 

Resource  Utilization 


CommsPost 


Averages  Half-Widths 

User  AnalystUser  AnalystUser 


Averages 

Half-Width 

Averages 

|o  |o 

1 

2 

3 

4 

5 

0.1908 

0.2057 

0.1957 

0.1995 

0.2099 

0.000256 

0.000278 

0.000257 

0.000240 

0.000262 

Source 

CommsCollectTX 

Processor  Src 

Analyst  Src 

AIISourceAnalyst  Spec 

InfoSource  1 

1.0007 

0.0121 

0.0226 

0.6743 

0.9838 

InfoSource  2 

1.0024 

0.0122 

0.0292 

0.5733 

0.9677 

InfoSource  3 

1.0047 

0.0149 

0.0363 

1.0162 

0.9574 

InfoSource  4 

1.0035 

0.0249 

0.0619 

0.9973 

0.9519 

Half-Widths 

Source 

CommsCollectTX 

Processor  Src 

Analyst  Src 

AIISourceAnalyst  Spec 

InfoSource  1 

0.000099 

0.000007 

0.000010 

0.000388 

0.000118 

InfoSource  2 

0.000056 

0.000004 

0.000004 

0.000219 

0.000236 

InfoSource  3 

0.000062 

0.000005 

0.000006 

0.000082 

0.000308 

InfoSource  4 

0.000047 

0.000010 

0.000016 

0.000118 

0.000316 

Figure  F.8  Case  2  simulation  results  part  2. 


Queue  Number  Waiting 

Averages 

Queue  Wait  Time 

Averages 

Wait  Time  by  Priority 

Averages 

Average 

Max 

Average  Max 

Priority  Average 

Max 

Search  Library 

3.05 

60.16 

0.8766  9.1206 

1 

3.06 

22.08 

Collect  Data 

349.87 

1152.00 

46.3907  1799.5178 

2 

10.29 

75.5 

TX  from  Collection 

0.00 

3.84 

0.0003  0.0979 

3 

38.73 

320.57 

Process  Data 

0.00 

3.36 

0.0008  0.0839 

4 

130.3 

1318.65 

Exploit  Data 

85.80 

286.76 

22.4632  3012.5203 

5 

311.74 

3091.8 

Analyze  Intel 

19.24 

140.64 

5.2179  392.5094 

Produce  Intel 

8.52 

80.32 

2.5045  170.2537 

Disseminate  Intel 

0.68 

19.84 

0.1205  9.4541 

Integrate  Intel 

0.00 

0.00 

=  = 

Half-Widths 

Half-Widths 

Half-Widths 

Average 

Max 

Average  Max 

Priority  Average 

Max 

Search  Library 

0.0040 

0.4841 

0.000497  0.083905 

1 

0.0025 

0.3078 

Collect  Data 

1 .7445 

15.0702 

0.200104  23.890979 

2 

0.0200 

0.7831 

TX  from  Collection 

0.0000 

0.0702 

0.000004  0.000912 

3 

0.0773 

3.2670 

Process  Data 

0.0000 

0.0469 

0.000005  0.001214 

4 

0.3977 

22.5456 

Exploit  Data 

0.6464 

4.3303 

0.160679  75.878188 

5 

1.4061 

75.8207 

Analyze  Intel 

0.1053 

1 .5529 

0.026361  3.833825 

Produce  Intel 

0.0338 

1.0107 

0.009557  2.636227 

Disseminate  Intel 

0.0011 

0.2180 

0.000196  0.167818 

Integrate  Intel 

0.0000 

0.0000 

=  = 

Figure  F.9  Case  2  simulation  results  part  3. 


F-6 


F.4  Case  3  Results 


alpha  0.05 

Requirements 

Averages 


Priority 

Timely 

QualMet 

Both 

User 

Timely 

QualMet 

Both 

1 

0.9984 

1.0000 

0.9984 

1 

0.6585 

0.9487 

0.6585 

2 

0.9699 

0.9999 

0.9699 

2 

0.7589 

0.9487 

0.7589 

3 

0.6291 

0.9931 

0.6291 

3 

0.7051 

0.9489 

0.7051 

4 

0.3337 

0.8456 

0.3337 

4 

0.7360 

0.9485 

0.7360 

5 

0.0996 

0.6997 

0.0996 

5 

0.7927 

0.9488 

0.7927 

Half-Widths 

Priority 

Timely 

QualMet 

Both 

User 

Timely 

QualMet 

Both 

1 

0.000012 

0.000001 

0.000012 

1 

0.000261 

0.000112 

0.000261 

2 

0.000130 

0.000003 

0.000130 

2 

0.000445 

0.000160 

0.000445 

3 

0.000887 

0.000055 

0.000887 

3 

0.000404 

0.000135 

0.000404 

4 

0.001091 

0.000602 

0.001091 

4 

0.000424 

0.000122 

0.000424 

5 

0.000577 

0.000572 

0.000577 

5 

0.000385 

0.000136 

0.000385 

Averages 

InfoSource 

Timely 

QualMet 

Both 

Timely 

QualMet 

Both 

1 

0.7464 

0.9473 

0.7464 

Additional 

0.8244 

0.9985 

0.8244 

2 

0.7752 

0.9482 

0.7752 

Standard 

0.6516 

0.9072 

0.6516 

3 

0.7022 

0.9509 

0.7022 

4 

0.7170 

0.9483 

0.7170 

Half-Widths 

InfoSource 

Timely 

QualMet 

Both 

Timely 

QualMet 

Both 

1 

0.000563 

0.000191 

0.000563 

Additional 

0.000377 

0.000017 

0.000377 

2 

0.000436 

0.000129 

0.000436 

Standard 

0.000409 

0.000220 

0.000409 

3 

0.000352 

0.000113 

0.000352 

4 

0.000445 

0.000148 

0.000445 

Figure  F.IO  Case  3  simulation  results  part  1. 


F-7 


WIP 

Averages 

Average 

Max 

1276  max(max) 

RFI 

370.58 

920.48 

718  min{max) 

Half-Widths 

Average 

Max 

RFI 

1.0255 

9.5987 

Resource  Utilization 

CommsPost 

Averages 

0 

Half-Width 


Averages 


Averages 


Half-Widths 


User  AnalystUser  AnalystUser 

1  0.1882  0.000258 

2  0.2038  0.000219 

3  0.1961  0.000206 

4  0.1989  0.000216 

5  0.2084  0.000266 


Source  CommsCollectTX  Processor_Src  Analyst_Src  AIISourceAnalyst_Spec 
InfoSourcel  0.9973  0.0121  0.0223  0.6677  0.9797 

lnfoSource2  1.0008  0.0121  0.0290  0.5712  0.9595 

lnfoSource3  1.0022  0.0149  0.0361  1.0131  0.9467 

lnfoSource4  1.0017  0.0249  0.0617  0.9936  0.9402 


Half-Widths 


Source  CommsCollectTX  Processor_Src  Analyst_Src  AIISourceAnalyst_Spec 


InfoSource  1 

0.000099 

0.000006 

0.000009  0.000419 

0.000086 

InfoSource  2 

0.000071 

0.000006 

0.000004  0.000228 

0.000187 

InfoSource  3 

0.000091 

0.000006 

0.000005  0.000078 

0.000251 

InfoSource  4 

0.000089 

0.000011 

0.000018  0.000123 

0.000290 

Figure  F.ll  Case  3  simulation  results  part  2. 

Queue  Number  Waiting 

Queue  Wait  Time 

Wait  Time  by  Priority 

Averages 

Averages 

Averages 

Average 

Max 

Average  Max 

Priority  Average  Max 

Search  Library 

3.05 

59.16 

0.8786  8.9236 

1 

2.98  20.82 

Collect  Data 

245.88 

805.44 

32.7077  1214.0334 

2 

9.61  67.86 

TX  from  Collection 

0.00 

3.64 

0.0003  0.0906 

3 

30.71  258.58 

Process  Data 

0.00 

3.36 

0.0007  0.0839 

4 

91.18  867.8 

Exploit  Data 

59.37 

206.28 

15.6284  1684.6015 

5 

206.57  1746.68 

Analyze  Intel 

14.42 

118.40 

3.9453  286.4975 

Produce  Intel 

7.20 

71.92 

2.1299  151.2150 

Disseminate  Intel 

0.67 

19.68 

0.1209  10.1141 

Integrate  Intel 

0.00 

0.00 

=  = 

Half-Widths 

Half-Widths 

Half-Widths 

Average 

Max 

Average  Max 

Priority  Average  Max 

Search  Library 

0.0031 

0.6727 

0.000556  0.124577 

1 

0.0015  0.2299 

Collect  Data 

1 .0529 

9.6125 

0.132673  20.079266 

2 

0.0106  0.6756 

TX  from  Collection 

0.0000 

0.0404 

0.000003  0.000894 

3 

0.0535  3.9127 

Process  Data 

0.0000 

0.0469 

0.000005  0.001037 

4 

0.2538  14.1835 

Exploit  Data 

0.2162 

1 .4740 

0.054748  25.128307 

5 

0.9477  24.2340 

Analyze  Intel 

0.0571 

1 .3578 

0.014454  4.086079 

Produce  Intel 

0.0220 

1 .3800 

0.006356  2.603467 

Disseminate  Intel 

0.0010 

0.2307 

0.000187  0.200957 

Integrate  Intel 

0.0000 

0.0000 

=  = 

Figure  F.12  Case  3  simulation  results  part  3. 


F-8 


F.5  Case  4  Results 


alpha  0.05 

Requirements 

Averages 


Priority 

Timely 

QualMet 

Both 

User 

Timely 

QualMet 

Both 

1 

0.9984 

1.0000 

0.9984 

1 

0.6549 

0.8725 

0.6549 

2 

0.9690 

0.9998 

0.9690 

2 

0.7565 

0.8725 

0.7565 

3 

0.6220 

0.9754 

0.6220 

3 

0.7019 

0.8729 

0.7019 

4 

0.3244 

0.6158 

0.3244 

4 

0.7317 

0.8726 

0.7317 

5 

0.0932 

0.2710 

0.0932 

5 

0.7894 

0.8729 

0.7894 

Half-Widths 

Priority 

Timely 

QualMet 

Both 

User 

Timely 

QualMet 

Both 

1 

0.000010 

0.000001 

0.000010 

1 

0.000393 

0.000320 

0.000393 

2 

0.000189 

0.000009 

0.000190 

2 

0.000356 

0.000234 

0.000355 

3 

0.000754 

0.000143 

0.000754 

3 

0.000320 

0.000298 

0.000320 

4 

0.001037 

0.001220 

0.001037 

4 

0.000335 

0.000281 

0.000335 

5 

0.000638 

0.001327 

0.000638 

5 

0.000444 

0.000321 

0.000443 

Averages 

InfoSource 

Timely 

QualMet 

Both 

Timely 

QualMet 

Both 

1 

0.7406 

0.8632 

0.7406 

Additional 

0.8216 

0.9924 

0.8215 

2 

0.7718 

0.8724 

0.7718 

Standard 

0.6476 

0.7725 

0.6476 

3 

0.7006 

0.8807 

0.7006 

4 

0.7138 

0.8728 

0.7138 

Half-Widths 

InfoSource 

Timely 

QualMet 

Both 

Timely 

QualMet 

Both 

1 

0.000545 

0.000372 

0.000545 

Additional 

0.000327 

0.000045 

0.000327 

2 

0.000432 

0.000289 

0.000432 

Standard 

0.000425 

0.000518 

0.000425 

3 

0.000388 

0.000312 

0.000388 

4 

0.000322 

0.000301 

0.000322 

Figure  F.13  Case  4  simulation  results  part  1. 


F-9 


WIP 

Averages 

Average 

Max 

1230  max(max) 

RFI 

381.31 

949.36 

809  min{max) 

Half-Widths 

Average 

Max 

RFI 

1.0522 

8.6682 

Resource  Utilization 

CommsPost 

Averages 

0 

Half-Width 


Averages 


Averages 


Half-Widths 


User  AnalystUser  AnalystUser 
1  0.1882  0.000266 

2  0.2032  0.000246 

3  0.1953  0.000320 

4  0.1999  0.000261 

5  0.2091  0.000316 


Source  CommsCollectTX  Processor_Src  Analyst_Src  AIISourceAnalyst_Spec 
0.9979  0.0121  0.0223  0.6677  0.9782 


InfoSource  1 


InfoSource  2 

1.0010 

0.0121 

0.0290 

0.5707 

0.9563 

InfoSource  3 

1.0026 

0.0149 

0.0361 

1.0128 

0.9428 

InfoSource  4 

1.0020 

0.0248 

0.0616 

0.9934 

0.9356 

Half-Widths 

Source  CommsCollectTX 

Processor  Src  Analyst  Src  AIISourceAnalyst  Spec 

InfoSource  1 

0.000090 

0.000008 

0.000009  0.000332 

0.000113 

InfoSource  2 

0.000074 

0.000005 

0.000004  0.000236 

0.000225 

InfoSource  3 

0.000094 

0.000005 

0.000006  0.000125 

0.000299 

InfoSource  4 

0.000078 

0.000009 

0.000017  0.000181 

0.000393 

Figure  F.14  Case  4  simulation  results  part  2. 

Queue  Number  Waiting 

Queue  Wait  Time 

Wait  Time  by  Priority 

Averages 

Averages 

Averages 

Average 

Max 

Average  Max 

Priority  Average  Max 

Search  Library 

3.05 

60.44 

0.8755  8.7620 

1 

2.98  21.39 

Collect  Data 

258.87 

828.68 

34.3492  1274.3049 

2 

9.69  67.25 

TX  from  Collection 

0.00 

3.60 

0.0003  0.0979 

3 

31.12  267.87 

Process  Data 

0.00 

3.48 

0.0007  0.0855 

4 

92.5  897.75 

Exploit  Data 

58.17 

205.08 

15.2752  1683.6870 

5 

217.43  1765.8 

Analyze  Intel 

13.74 

112.36 

3.7639  263.0930 

Produce  Intel 

6.85 

68.68 

2.0384  137.8200 

Disseminate  Intel 

0.67 

19.00 

0.1205  9.9034 

Integrate  Intel 

0.00 

0.00 

=  = 

Half-Widths 

Half-Widths 

Half-Widths 

Average 

Max 

Average  Max 

Priority  Average  Max 

Search  Library 

0.0033 

0.6754 

0.000400  0.117231 

1 

0.0021  0.1792 

Collect  Data 

1.1208 

7.9886 

0.136397  15.285306 

2 

0.0135  0.5753 

TX  from  Collection 

0.0000 

0.0477 

0.000003  0.000848 

3 

0.0414  4.6343 

Process  Data 

0.0000 

0.0421 

0.000005  0.000965 

4 

0.2280  17.6601 

Exploit  Data 

0.2967 

1 .5637 

0.070643  36.890073 

5 

0.9201  35.1022 

Analyze  Intel 

0.0542 

1.2131 

0.013363  3.542768 

Produce  Intel 

0.0209 

0.7795 

0.005781  2.553720 

Disseminate  Intel 

0.0010 

0.2347 

0.000186  0.161694 

Integrate  Intel 

0.0000 

0.0000 

=  = 

Figure  F.15  Case  4  simulation  results  part  3. 


F-10 


F.6  Case  5  Results 


alpha  0.05 

Requirements 

Averages 


Priority 

Timely 

QualMet 

Both 

User 

Timely 

QualMet 

Both 

1 

0.9974 

0.9688 

0.9668 

1  0.5112 

0.7744 

0.5028 

2 

0.7858 

0.9580 

0.7616 

2  0.5992 

0.7493 

0.5722 

3 

0.2240 

0.7944 

0.2167 

3  0.5536 

0.7722 

0.5435 

4 

0.0510 

0.1069 

0.0485 

4  0.5841 

0.7773 

0.5772 

5 

0.0071 

0.0172 

0.0067 

5  0.6313 

0.7416 

0.5941 

Half-Widths 

Priority 

Timely 

QualMet 

Both 

User 

Timely 

QualMet 

Both 

1 

0.000012 

0.000038 

0.000037 

1  0.000437 

0.000363 

0.000430 

2 

0.000809 

0.000111 

0.000791 

2  0.000457 

0.000394 

0.000422 

3 

0.000781 

0.000494 

0.000760 

3  0.000498 

0.000416 

0.000488 

4 

0.000480 

0.000898 

0.000462 

4  0.000435 

0.000415 

0.000430 

5 

0.000155 

0.000299 

0.000146 

5  0.000513 

0.000478 

0.000483 

Averages 

InfoSource 

Timely 

QualMet 

Both 

Timely 

QualMet 

Both 

1 

0.5771 

0.5984 

0.4830 

Additional 

0.6424 

0.9292 

0.6228 

2 

0.6080 

0.7894 

0.6080 

Standard 

0.4911 

0.5514 

0.4753 

3 

0.5575 

0.8045 

0.5575 

4 

0.5703 

0.8060 

0.5703 

Half-Widths 

InfoSource 

Timely 

QualMet 

Both 

Timely 

QualMet 

Both 

1 

0.000478 

0.000378 

0.000411 

Additional 

0.000441 

0.000086 

0.000427 

2 

0.000553 

0.000465 

0.000553 

Standard 

0.000608 

0.000682 

0.000589 

3 

0.000478 

0.000359 

0.000478 

4 

0.000560 

0.000604 

0.000560 

Figure 

F.16 

Case  5 

simulation  results  part  1. 

F-11 


WIP 

Averages 

Average 

Max 

7526  max(max) 

RFI 

Half-Widths 

2123.01 

5269.68 

3279  min{max) 

Average 

Max 

RFI 

27.3718 

85.5808 

Resource  Utilization 


Averages 


Half-Widths 


CommsPost 

User  AnalystUser 

Analyst  User 

Averages 

1 

0.2312 

0.000347 

0 

2 

0.2454 

0.000388 

Half-Width 

3 

0.2370 

0.000436 

0 

4 

0.2445 

0.000321 

5 

0.2514 

0.000428 

Averages 

Source  CommsCollectTX 

Processor  Src 

Analyst  Src  AIISourceAnalyst  Spec 

InfoSource  1 

1.0044 

0.0122 

0.0221 

0.6589 

0.9681 

InfoSource  2 

1.0039 

0.0122 

0.0286 

0.5633 

0.9352 

InfoSource  3 

1.0069 

0.0149 

0.0361 

1.0047 

0.9164 

InfoSource  4 

1.0056 

0.0249 

0.0616 

0.9831 

0.9063 

Half-Widths 

Source  CommsCollectTX 

Processor  Src 

Analyst  Src  AIISourceAnalyst  Spec 

InfoSource  1 

0.000020 

0.000006 

0.000009 

0.000387 

0.000139 

InfoSource  2 

0.000010 

0.000005 

0.000004 

0.000353 

0.000271 

InfoSource  3 

0.000016 

0.000007 

0.000005 

0.000144 

0.000335 

InfoSource  4 

0.000018 

0.000011 

0.000017 

0.000177 

0.000355 

Figure  F.17 

Case 

5  simulation  results  part  2. 

Queue  Number  Waiting 

Averages 

Queue  Wait  Time 

Averages 

Wait  Time  by  Priority 

Averages 

Average 

Max 

Average 

Max 

Priority  Average 

Max 

Search  Library 

7.69 

107.00 

1.4555 

15.9069 

1 

4.23 

31.42 

Collect  Data 

2010.36 

5158.88 

207.9107 

8641 .5306 

2 

20.12 

139.46 

TX  from  Collection 

0.01 

4.80 

0.0008 

0.1149 

3 

66.48 

625.05 

Process  Data 
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Figure  F.18  Case  5  simulation  results  part  3. 
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F.7  Case  6  Results 
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Figure  F.19  Case  6  simulation  results  part  1. 
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Figure  F.21  Case  6  simulation  results  part  3. 
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F.8  Case  7  Results 
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Figure  F.22  Case  7  simulation  results  part  1. 
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Figure  F.23 
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Figure  F.24  Case  7  simulation  results  part  3. 
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F.9 


Selected  Paired-t  tests 
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Figure  F.25  p-values  for  selected  paired-t  tests  of  User  1  and  User  5. 
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